Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X....X..................X...............X...........X....] duration:
2.88838795s trace:
https://jaeger.dev.storj.io/trace/5f0044a50b568df8
FAIL (2025-10-31 21:24:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.888414931s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................X.........................XX..] duration:
2.346574631s trace:
https://jaeger.dev.storj.io/trace/3173f13c3ae66d35
FAIL (2025-10-31 21:24:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
NOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.439072473 (33h41m54.140664943s) [noise=true , address=145.79.227.33:4888 , piece_id=55HIHYQGP4PYV47T3EOPUOOWH74Q2AC66ZJAVNMNY2NUDV5DM7QA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.440028262 (33h41m54.141620722s) [noise=true , address=145.79.227.36:4549 , piece_id=OPMGCVNCWD44POIDEO46A4ZKUJ4BQE7UDPSIF3UAIDWZP2QJRC4A, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.440879298 (33h41m54.142471755s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.441210726 (33h41m54.142803185s) [noise=true , address=77.90.31.19:4865 , piece_id=ZDUGAOKKTEBBUF4S7ZFOGSBITGLV5BUW2RHCVHVSDVPEYNTEWDOQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.441758919 (33h41m54.143351377s) [noise=true , address=77.90.31.12:4609 , piece_id=N4LUJ74AFB7JAE4ZL66HLKG746LGHKNFBY7TZTSTQOTIBPQN7I2Q, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442265638 (33h41m54.143858095s) [noise=true , address=45.91.116.10:4113 , piece_id=PIGRIKVRZCWODXJVZ4SIORYLZYWRMMB2EXQRZGJZRK3JCXLWRKOA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442303271 (33h41m54.14389573s) [noise=true , address=77.90.31.21:4910 , piece_id=K4ERYFVADFU242IHSKR6BGQJHLUNSQPIHNFHPA7QLNF7JSJSACZQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442656746 (33h41m54.144249202s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442782370 (33h41m54.144374829s) [noise=true , address=45.91.116.9:4286 , piece_id=3T3JM6QB6GN4OUSP2LN7ICHBZL3CO5GW56ODFBB7KVFNXSHRD52A, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443021968 (33h41m54.144614421s) [noise=true , address=45.91.116.8:4057 , piece_id=BVGJB2U2MHIEUOUOPF5GF6SY63DVJQ54LLI4W7SND3DZX56MZZUQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443252911 (33h41m54.144845368s) [noise=true , address=77.90.31.21:4911 , piece_id=FMIATBPBQP237AJC3IMMG7KC74M424BQ2KYSXJQOZHUMIIXXAVWQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443438074 (33h41m54.145030531s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443753610 (33h41m54.145346067s) [noise=true , address=145.79.227.6:4269 , piece_id=ZEEU26BQ4H4WTK4TMHGJYQPTQMQNQOWJGAECEINQCQDKTKATZ7DA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443851896 (33h41m54.145444348s) [noise=true , address=45.91.116.5:4308 , piece_id=FTWXK442NYHAARIK7REHW7GHHQ6K4URWJD75K6QNYXCI3S3BMKWA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.444315561 (33h41m54.145908018s) [noise=true , address=45.91.116.12:4242 , piece_id=TXGM7MH6RQ33AHDFWD4X3DSNZGL4DP4WBEFUFQXW5HEUPE56JWYA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.445026431 (33h41m54.14661889s) [noise=true , address=45.91.116.13:4177 , piece_id=XREU7IAJVWS2MJIEEEFOVKQOIAXJ3DGSX3NDA7KJUL6IUIXMX4PQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.445715106 (33h41m54.147307563s) [noise=true , address=145.79.227.52:4461 , piece_id=LQJVVKNJNZOVEXRW7Y5KGLX4IDFZMCG65UN3BAVPJEIHWN3NIWQA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:52.002260082 (33h41m54.703852532s) [upload=29647]: Next returned nil segment
Oct 31 21:22:52.002281375 (33h41m54.703873836s) [upload=29647]: Waiting for error group managing segments...
Oct 31 21:22:52.166910441 (33h41m54.868502889s) [noise=true , address=145.79.227.33:4888 , piece_id=55HIHYQGP4PYV47T3EOPUOOWH74Q2AC66ZJAVNMNY2NUDV5DM7QA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.171085961 (33h41m54.872678409s) [noise=true , address=145.79.227.36:4549 , piece_id=OPMGCVNCWD44POIDEO46A4ZKUJ4BQE7UDPSIF3UAIDWZP2QJRC4A, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.173190935 (33h41m54.874783383s) [noise=true , address=145.79.227.52:4461 , piece_id=LQJVVKNJNZOVEXRW7Y5KGLX4IDFZMCG65UN3BAVPJEIHWN3NIWQA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.174554840 (33h41m54.876147316s) [noise=true , address=145.79.227.6:4269 , piece_id=ZEEU26BQ4H4WTK4TMHGJYQPTQMQNQOWJGAECEINQCQDKTKATZ7DA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.870199575 (33h41m55.571792024s) [noise=true , address=45.91.116.8:4057 , piece_id=BVGJB2U2MHIEUOUOPF5GF6SY63DVJQ54LLI4W7SND3DZX56MZZUQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.932896280 (33h41m55.634488728s) [noise=true , address=45.91.116.5:4308 , piece_id=FTWXK442NYHAARIK7REHW7GHHQ6K4URWJD75K6QNYXCI3S3BMKWA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.996436348 (33h41m55.698028803s) [noise=true , address=45.91.116.13:4177 , piece_id=XREU7IAJVWS2MJIEEEFOVKQOIAXJ3DGSX3NDA7KJUL6IUIXMX4PQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:53.303664204 (33h41m56.005256643s) [noise=true , address=45.91.116.10:4113 , piece_id=PIGRIKVRZCWODXJVZ4SIORYLZYWRMMB2EXQRZGJZRK3JCXLWRKOA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:53.934904546 (33h41m56.636497004s) [noise=true , address=77.90.31.21:4910 , piece_id=K4ERYFVADFU242IHSKR6BGQJHLUNSQPIHNFHPA7QLNF7JSJSACZQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:54.224811350 (33h41m56.926403798s) [noise=true , address=77.90.31.12:4609 , piece_id=N4LUJ74AFB7JAE4ZL66HLKG746LGHKNFBY7TZTSTQOTIBPQN7I2Q, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:54.652552655 (33h41m57.35414511s) [noise=true , address=45.91.116.9:4286 , piece_id=3T3JM6QB6GN4OUSP2LN7ICHBZL3CO5GW56ODFBB7KVFNXSHRD52A, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:55.813839187 (33h41m58.515431634s) [noise=true , address=45.91.116.12:4242 , piece_id=TXGM7MH6RQ33AHDFWD4X3DSNZGL4DP4WBEFUFQXW5HEUPE56JWYA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:56.776728453 (33h41m59.4783209s) [noise=true , address=77.90.31.21:4911 , piece_id=FMIATBPBQP237AJC3IMMG7KC74M424BQ2KYSXJQOZHUMIIXXAVWQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:58.637065006 (33h42m1.338657444s) [noise=true , address=77.90.31.19:4865 , piece_id=ZDUGAOKKTEBBUF4S7ZFOGSBITGLV5BUW2RHCVHVSDVPEYNTEWDOQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:58.637090306 (33h42m1.338682758s) [seg_pos={0 0}, upload=29647]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:22:58.637316080 (33h42m1.338908539s) [noise=true , address=77.90.31.17:4787 , piece_id=V3EHNOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637326708 (33h42m1.338919172s) [noise=true , address=77.90.31.17:4787 , piece_id=V3EHNOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637380935 (33h42m1.338973393s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637394866 (33h42m1.338987321s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637446506 (33h42m1.339038963s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637456777 (33h42m1.339049231s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637522129 (33h42m1.339114607s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637528827 (33h42m1.339121275s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637546736 (33h42m1.339139173s) [seg_pos={0 0}, upload=29647]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:22:58.637649415 (33h42m1.339241873s) [upload=29647]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:22:58.637657672 (33h42m1.339250145s) [upload=29647]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:22:58.787848722 (33h42m1.489441173s) [upload=29647]: Done uploading segments...
Oct 31 21:22:58.787868799 (33h42m1.489461251s) [upload=29647]: Upload finished. err: <nil>
Duration: 2.346602397s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............................X.........................XX.] duration:
33.240385524s trace:
https://jaeger.dev.storj.io/trace/3173f13c3ae66d35
FAIL (2025-10-31 21:24:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
NOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.439072473 (33h41m54.140664943s) [noise=true , address=145.79.227.33:4888 , piece_id=55HIHYQGP4PYV47T3EOPUOOWH74Q2AC66ZJAVNMNY2NUDV5DM7QA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.440028262 (33h41m54.141620722s) [noise=true , address=145.79.227.36:4549 , piece_id=OPMGCVNCWD44POIDEO46A4ZKUJ4BQE7UDPSIF3UAIDWZP2QJRC4A, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.440879298 (33h41m54.142471755s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.441210726 (33h41m54.142803185s) [noise=true , address=77.90.31.19:4865 , piece_id=ZDUGAOKKTEBBUF4S7ZFOGSBITGLV5BUW2RHCVHVSDVPEYNTEWDOQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.441758919 (33h41m54.143351377s) [noise=true , address=77.90.31.12:4609 , piece_id=N4LUJ74AFB7JAE4ZL66HLKG746LGHKNFBY7TZTSTQOTIBPQN7I2Q, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442265638 (33h41m54.143858095s) [noise=true , address=45.91.116.10:4113 , piece_id=PIGRIKVRZCWODXJVZ4SIORYLZYWRMMB2EXQRZGJZRK3JCXLWRKOA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442303271 (33h41m54.14389573s) [noise=true , address=77.90.31.21:4910 , piece_id=K4ERYFVADFU242IHSKR6BGQJHLUNSQPIHNFHPA7QLNF7JSJSACZQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442656746 (33h41m54.144249202s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442782370 (33h41m54.144374829s) [noise=true , address=45.91.116.9:4286 , piece_id=3T3JM6QB6GN4OUSP2LN7ICHBZL3CO5GW56ODFBB7KVFNXSHRD52A, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443021968 (33h41m54.144614421s) [noise=true , address=45.91.116.8:4057 , piece_id=BVGJB2U2MHIEUOUOPF5GF6SY63DVJQ54LLI4W7SND3DZX56MZZUQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443252911 (33h41m54.144845368s) [noise=true , address=77.90.31.21:4911 , piece_id=FMIATBPBQP237AJC3IMMG7KC74M424BQ2KYSXJQOZHUMIIXXAVWQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443438074 (33h41m54.145030531s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443753610 (33h41m54.145346067s) [noise=true , address=145.79.227.6:4269 , piece_id=ZEEU26BQ4H4WTK4TMHGJYQPTQMQNQOWJGAECEINQCQDKTKATZ7DA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443851896 (33h41m54.145444348s) [noise=true , address=45.91.116.5:4308 , piece_id=FTWXK442NYHAARIK7REHW7GHHQ6K4URWJD75K6QNYXCI3S3BMKWA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.444315561 (33h41m54.145908018s) [noise=true , address=45.91.116.12:4242 , piece_id=TXGM7MH6RQ33AHDFWD4X3DSNZGL4DP4WBEFUFQXW5HEUPE56JWYA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.445026431 (33h41m54.14661889s) [noise=true , address=45.91.116.13:4177 , piece_id=XREU7IAJVWS2MJIEEEFOVKQOIAXJ3DGSX3NDA7KJUL6IUIXMX4PQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.445715106 (33h41m54.147307563s) [noise=true , address=145.79.227.52:4461 , piece_id=LQJVVKNJNZOVEXRW7Y5KGLX4IDFZMCG65UN3BAVPJEIHWN3NIWQA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:52.002260082 (33h41m54.703852532s) [upload=29647]: Next returned nil segment
Oct 31 21:22:52.002281375 (33h41m54.703873836s) [upload=29647]: Waiting for error group managing segments...
Oct 31 21:22:52.166910441 (33h41m54.868502889s) [noise=true , address=145.79.227.33:4888 , piece_id=55HIHYQGP4PYV47T3EOPUOOWH74Q2AC66ZJAVNMNY2NUDV5DM7QA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.171085961 (33h41m54.872678409s) [noise=true , address=145.79.227.36:4549 , piece_id=OPMGCVNCWD44POIDEO46A4ZKUJ4BQE7UDPSIF3UAIDWZP2QJRC4A, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.173190935 (33h41m54.874783383s) [noise=true , address=145.79.227.52:4461 , piece_id=LQJVVKNJNZOVEXRW7Y5KGLX4IDFZMCG65UN3BAVPJEIHWN3NIWQA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.174554840 (33h41m54.876147316s) [noise=true , address=145.79.227.6:4269 , piece_id=ZEEU26BQ4H4WTK4TMHGJYQPTQMQNQOWJGAECEINQCQDKTKATZ7DA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.870199575 (33h41m55.571792024s) [noise=true , address=45.91.116.8:4057 , piece_id=BVGJB2U2MHIEUOUOPF5GF6SY63DVJQ54LLI4W7SND3DZX56MZZUQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.932896280 (33h41m55.634488728s) [noise=true , address=45.91.116.5:4308 , piece_id=FTWXK442NYHAARIK7REHW7GHHQ6K4URWJD75K6QNYXCI3S3BMKWA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.996436348 (33h41m55.698028803s) [noise=true , address=45.91.116.13:4177 , piece_id=XREU7IAJVWS2MJIEEEFOVKQOIAXJ3DGSX3NDA7KJUL6IUIXMX4PQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:53.303664204 (33h41m56.005256643s) [noise=true , address=45.91.116.10:4113 , piece_id=PIGRIKVRZCWODXJVZ4SIORYLZYWRMMB2EXQRZGJZRK3JCXLWRKOA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:53.934904546 (33h41m56.636497004s) [noise=true , address=77.90.31.21:4910 , piece_id=K4ERYFVADFU242IHSKR6BGQJHLUNSQPIHNFHPA7QLNF7JSJSACZQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:54.224811350 (33h41m56.926403798s) [noise=true , address=77.90.31.12:4609 , piece_id=N4LUJ74AFB7JAE4ZL66HLKG746LGHKNFBY7TZTSTQOTIBPQN7I2Q, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:54.652552655 (33h41m57.35414511s) [noise=true , address=45.91.116.9:4286 , piece_id=3T3JM6QB6GN4OUSP2LN7ICHBZL3CO5GW56ODFBB7KVFNXSHRD52A, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:55.813839187 (33h41m58.515431634s) [noise=true , address=45.91.116.12:4242 , piece_id=TXGM7MH6RQ33AHDFWD4X3DSNZGL4DP4WBEFUFQXW5HEUPE56JWYA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:56.776728453 (33h41m59.4783209s) [noise=true , address=77.90.31.21:4911 , piece_id=FMIATBPBQP237AJC3IMMG7KC74M424BQ2KYSXJQOZHUMIIXXAVWQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:58.637065006 (33h42m1.338657444s) [noise=true , address=77.90.31.19:4865 , piece_id=ZDUGAOKKTEBBUF4S7ZFOGSBITGLV5BUW2RHCVHVSDVPEYNTEWDOQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:58.637090306 (33h42m1.338682758s) [seg_pos={0 0}, upload=29647]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:22:58.637316080 (33h42m1.338908539s) [noise=true , address=77.90.31.17:4787 , piece_id=V3EHNOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637326708 (33h42m1.338919172s) [noise=true , address=77.90.31.17:4787 , piece_id=V3EHNOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637380935 (33h42m1.338973393s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637394866 (33h42m1.338987321s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637446506 (33h42m1.339038963s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637456777 (33h42m1.339049231s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637522129 (33h42m1.339114607s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637528827 (33h42m1.339121275s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637546736 (33h42m1.339139173s) [seg_pos={0 0}, upload=29647]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:22:58.637649415 (33h42m1.339241873s) [upload=29647]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:22:58.637657672 (33h42m1.339250145s) [upload=29647]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:22:58.787848722 (33h42m1.489441173s) [upload=29647]: Done uploading segments...
Oct 31 21:22:58.787868799 (33h42m1.489461251s) [upload=29647]: Upload finished. err: <nil>
Duration: 33.240417643s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
2.526184174s trace:
https://jaeger.dev.storj.io/trace/62df97e1df2d7a
FAIL (2025-10-31 21:24:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:22:07.085611190 (33h41m9.787203641s) [upload=29636]: Starting upload
Oct 31 21:22:07.085688329 (33h41m9.787280767s) [upload=29636]: Done starting upload
Oct 31 21:22:07.085774737 (33h41m9.78736719s) [upload=29636]: Uploading segments...
Oct 31 21:22:07.086049239 (33h41m9.787641687s) [upload=29636]: Got next segment. Inline: false
Oct 31 21:22:07.086139291 (33h41m9.78773174s) [upload=29636]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:07.086145090 (33h41m9.787737538s) [upload=29636]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:07.198887636 (33h41m9.900480077s) [seg_pos={0 0}, upload=29636]: Begin upload segment...
Oct 31 21:22:07.198943347 (33h41m9.900535799s) [seg_pos={0 0}, upload=29636]: Done begin upload segment.
Oct 31 21:22:07.198976223 (33h41m9.900568674s) [noise=true , address=145.79.227.27:4689 , piece_id=E2674435DTU7SSPTAFGMRLBKZNPZAOIFMBDN5N734CP4RG23SVCA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199094043 (33h41m9.900686499s) [noise=true , address=145.79.227.15:4875 , piece_id=GXNV3O7D7URXRTLWVXMFMVTXGHZ4PAZMGQLI3L5SNUX3LDV7OXJA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199764319 (33h41m9.901356931s) [noise=true , address=145.79.227.14:4840 , piece_id=MKR2JQOAFU6BA6JUG4LTAICYEDRI367JLXCNW6EPZWIJR2ZNKXYA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199979140 (33h41m9.901571602s) [noise=true , address=145.79.227.28:4725 , piece_id=2UEZO2DIIDOVE4V6TXLKZAWXTOUNTIJKCBOP6JHREEEYBUWEXAUA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.201094736 (33h41m9.902687196s) [noise=true , address=145.79.227.10:4401 , piece_id=YCZ6LL4ECE2QKCLEAF77L3C72VWCZ6IQVJLJ77RIACX4LIK2IJJA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.201150898 (33h41m9.902743361s) [noise=true , address=145.79.227.32:4834 , piece_id=LBXTDM2QLBWFNDJ5272FY546TCRWC3IFYRNOBCCIS3KQDFMVJDFQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.202300922 (33h41m9.903893383s) [noise=true , address=45.91.116.4:4940 , piece_id=7TW5URZLYHRH7MQX3PZGMYPKJS3SYTXCJJDSKCOQZHB2HIS6J4OA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.202347708 (33h41m9.903940165s) [noise=true , address=77.90.31.12:4590 , piece_id=WQQC3535ERLWQOD6WCWJF3DUY4FY4CZF7KPD6Y2K3Q2NYQCP45JA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.203610241 (33h41m9.905202697s) [noise=true , address=145.79.227.31:4798 , piece_id=QIYHHYEZGJEMENOVRJIDH7722MX2HI7XAGWUSFKIV3YNHKVPVD2Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.204358358 (33h41m9.905950816s) [noise=true , address=145.79.227.9:4367 , piece_id=Q2S6IK6OE4535GKC6J55MV3N45EUDWGOYKWS3ISQSZJKW2WJCD6Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.204958129 (33h41m9.906550588s) [noise=true , address=45.91.116.4:4939 , piece_id=VYJO7TDD6SANMOZ7BB62KYLU72ACLUIZI7IHFV6I2PDAV7K7WLCQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.206076686 (33h41m9.907669138s) [noise=true , address=145.79.227.30:4763 , piece_id=ET72Q2LHTA3HQY2GQNCO2ZEOBP526EOWM72TL5XRQYOLLARFPZLQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.207450031 (33h41m9.909042495s) [noise=true , address=45.91.116.8:4045 , piece_id=B6C3QMJBWPBBJBTSYFP67R5WOVVGIGOYN6B6P5BAW35FXYEVPW3A, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.208460871 (33h41m9.910053312s) [noise=true , address=145.79.227.15:4873 , piece_id=FIIZHJVUTZ42TATHLHZEWSRMVUFB7DYTWGIABNR4LJQYVH4SKQ5A, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.209461830 (33h41m9.911054339s) [noise=true , address=145.79.227.11:4729 , piece_id=ZYPWL4SEIZHXYOAW4I5JOCPXDTHRCMO2FXYGA4ZUNHA2CSZ6DG5Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.210046720 (33h41m9.911639172s) [noise=true , address=145.79.227.30:4762 , piece_id=4BL5PJ3S2OC7PVSILA4IQCFT4JW6W5GBOXI7WRAS52DNBCF3DIHQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.808217579 (33h41m10.509810059s) [upload=29636]: Next returned nil segment
Oct 31 21:22:07.808236009 (33h41m10.509828464s) [upload=29636]: Waiting for error group managing segments...
Oct 31 21:22:07.951052914 (33h41m10.65264537s) [noise=true , address=145.79.227.32:4834 , piece_id=LBXTDM2QLBWFNDJ5272FY546TCRWC3IFYRNOBCCIS3KQDFMVJDFQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.951804679 (33h41m10.653397134s) [noise=true , address=145.79.227.15:4875 , piece_id=GXNV3O7D7URXRTLWVXMFMVTXGHZ4PAZMGQLI3L5SNUX3LDV7OXJA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.951868872 (33h41m10.653461324s) [noise=true , address=145.79.227.31:4798 , piece_id=QIYHHYEZGJEMENOVRJIDH7722MX2HI7XAGWUSFKIV3YNHKVPVD2Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952470935 (33h41m10.654063385s) [noise=true , address=145.79.227.30:4762 , piece_id=4BL5PJ3S2OC7PVSILA4IQCFT4JW6W5GBOXI7WRAS52DNBCF3DIHQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952500937 (33h41m10.654093387s) [noise=true , address=145.79.227.27:4689 , piece_id=E2674435DTU7SSPTAFGMRLBKZNPZAOIFMBDN5N734CP4RG23SVCA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952980719 (33h41m10.654573168s) [noise=true , address=145.79.227.28:4725 , piece_id=2UEZO2DIIDOVE4V6TXLKZAWXTOUNTIJKCBOP6JHREEEYBUWEXAUA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.953254355 (33h41m10.654846814s) [noise=true , address=145.79.227.11:4729 , piece_id=ZYPWL4SEIZHXYOAW4I5JOCPXDTHRCMO2FXYGA4ZUNHA2CSZ6DG5Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.953604050 (33h41m10.6551965s) [noise=true , address=145.79.227.15:4873 , piece_id=FIIZHJVUTZ42TATHLHZEWSRMVUFB7DYTWGIABNR4LJQYVH4SKQ5A, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.954075643 (33h41m10.6556681s) [noise=true , address=145.79.227.9:4367 , piece_id=Q2S6IK6OE4535GKC6J55MV3N45EUDWGOYKWS3ISQSZJKW2WJCD6Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.958770427 (33h41m10.660362881s) [noise=true , address=145.79.227.10:4401 , piece_id=YCZ6LL4ECE2QKCLEAF77L3C72VWCZ6IQVJLJ77RIACX4LIK2IJJA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.965022314 (33h41m10.666614767s) [noise=true , address=145.79.227.30:4763 , piece_id=ET72Q2LHTA3HQY2GQNCO2ZEOBP526EOWM72TL5XRQYOLLARFPZLQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.976650683 (33h41m10.678243138s) [noise=true , address=145.79.227.14:4840 , piece_id=MKR2JQOAFU6BA6JUG4LTAICYEDRI367JLXCNW6EPZWIJR2ZNKXYA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:36.799544661 (33h41m39.501137117s) [noise=true , address=45.91.116.4:4939 , piece_id=VYJO7TDD6SANMOZ7BB62KYLU72ACLUIZI7IHFV6I2PDAV7K7WLCQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection
Oct 31 21:22:36.799605331 (33h41m39.50119779s) [noise=true , address=77.90.31.12:4590 , piece_id=WQQC3535ERLWQOD6WCWJF3DUY4FY4CZF7KPD6Y2K3Q2NYQCP45JA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection; write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection
Oct 31 21:22:36.799747990 (33h41m39.501340451s) [noise=true , address=45.91.116.4:4940 , piece_id=7TW5URZLYHRH7MQX3PZGMYPKJS3SYTXCJJDSKCOQZHB2HIS6J4OA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 139.59.144.22:36232->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:36.799846589 (33h41m39.501439048s) [noise=true , address=45.91.116.8:4045 , piece_id=B6C3QMJBWPBBJBTSYFP67R5WOVVGIGOYN6B6P5BAW35FXYEVPW3A, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:39300->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:36.799901279 (33h41m39.501493715s) [seg_pos={0 0}, upload=29636]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:22:36.906137907 (33h41m39.607730362s) [upload=29636]: Done uploading segments...
Oct 31 21:22:36.906171689 (33h41m39.60776414s) [upload=29636]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.526211125s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [...X....XX...................X..............X.........X.....] duration:
1.635311901s trace:
https://jaeger.dev.storj.io/trace/5b5403da1e956ee2
FAIL (2025-10-31 21:24:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.635340093s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [....X....XX...................X..............X.........X....] duration:
2.31217001s trace:
https://jaeger.dev.storj.io/trace/5b5403da1e956ee2
FAIL (2025-10-31 21:24:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.312197249s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
4.069207469s trace:
https://jaeger.dev.storj.io/trace/62df97e1df2d7a
FAIL (2025-10-31 21:24:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:22:07.085611190 (33h41m9.787203641s) [upload=29636]: Starting upload
Oct 31 21:22:07.085688329 (33h41m9.787280767s) [upload=29636]: Done starting upload
Oct 31 21:22:07.085774737 (33h41m9.78736719s) [upload=29636]: Uploading segments...
Oct 31 21:22:07.086049239 (33h41m9.787641687s) [upload=29636]: Got next segment. Inline: false
Oct 31 21:22:07.086139291 (33h41m9.78773174s) [upload=29636]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:07.086145090 (33h41m9.787737538s) [upload=29636]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:07.198887636 (33h41m9.900480077s) [seg_pos={0 0}, upload=29636]: Begin upload segment...
Oct 31 21:22:07.198943347 (33h41m9.900535799s) [seg_pos={0 0}, upload=29636]: Done begin upload segment.
Oct 31 21:22:07.198976223 (33h41m9.900568674s) [noise=true , address=145.79.227.27:4689 , piece_id=E2674435DTU7SSPTAFGMRLBKZNPZAOIFMBDN5N734CP4RG23SVCA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199094043 (33h41m9.900686499s) [noise=true , address=145.79.227.15:4875 , piece_id=GXNV3O7D7URXRTLWVXMFMVTXGHZ4PAZMGQLI3L5SNUX3LDV7OXJA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199764319 (33h41m9.901356931s) [noise=true , address=145.79.227.14:4840 , piece_id=MKR2JQOAFU6BA6JUG4LTAICYEDRI367JLXCNW6EPZWIJR2ZNKXYA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199979140 (33h41m9.901571602s) [noise=true , address=145.79.227.28:4725 , piece_id=2UEZO2DIIDOVE4V6TXLKZAWXTOUNTIJKCBOP6JHREEEYBUWEXAUA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.201094736 (33h41m9.902687196s) [noise=true , address=145.79.227.10:4401 , piece_id=YCZ6LL4ECE2QKCLEAF77L3C72VWCZ6IQVJLJ77RIACX4LIK2IJJA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.201150898 (33h41m9.902743361s) [noise=true , address=145.79.227.32:4834 , piece_id=LBXTDM2QLBWFNDJ5272FY546TCRWC3IFYRNOBCCIS3KQDFMVJDFQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.202300922 (33h41m9.903893383s) [noise=true , address=45.91.116.4:4940 , piece_id=7TW5URZLYHRH7MQX3PZGMYPKJS3SYTXCJJDSKCOQZHB2HIS6J4OA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.202347708 (33h41m9.903940165s) [noise=true , address=77.90.31.12:4590 , piece_id=WQQC3535ERLWQOD6WCWJF3DUY4FY4CZF7KPD6Y2K3Q2NYQCP45JA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.203610241 (33h41m9.905202697s) [noise=true , address=145.79.227.31:4798 , piece_id=QIYHHYEZGJEMENOVRJIDH7722MX2HI7XAGWUSFKIV3YNHKVPVD2Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.204358358 (33h41m9.905950816s) [noise=true , address=145.79.227.9:4367 , piece_id=Q2S6IK6OE4535GKC6J55MV3N45EUDWGOYKWS3ISQSZJKW2WJCD6Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.204958129 (33h41m9.906550588s) [noise=true , address=45.91.116.4:4939 , piece_id=VYJO7TDD6SANMOZ7BB62KYLU72ACLUIZI7IHFV6I2PDAV7K7WLCQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.206076686 (33h41m9.907669138s) [noise=true , address=145.79.227.30:4763 , piece_id=ET72Q2LHTA3HQY2GQNCO2ZEOBP526EOWM72TL5XRQYOLLARFPZLQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.207450031 (33h41m9.909042495s) [noise=true , address=45.91.116.8:4045 , piece_id=B6C3QMJBWPBBJBTSYFP67R5WOVVGIGOYN6B6P5BAW35FXYEVPW3A, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.208460871 (33h41m9.910053312s) [noise=true , address=145.79.227.15:4873 , piece_id=FIIZHJVUTZ42TATHLHZEWSRMVUFB7DYTWGIABNR4LJQYVH4SKQ5A, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.209461830 (33h41m9.911054339s) [noise=true , address=145.79.227.11:4729 , piece_id=ZYPWL4SEIZHXYOAW4I5JOCPXDTHRCMO2FXYGA4ZUNHA2CSZ6DG5Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.210046720 (33h41m9.911639172s) [noise=true , address=145.79.227.30:4762 , piece_id=4BL5PJ3S2OC7PVSILA4IQCFT4JW6W5GBOXI7WRAS52DNBCF3DIHQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.808217579 (33h41m10.509810059s) [upload=29636]: Next returned nil segment
Oct 31 21:22:07.808236009 (33h41m10.509828464s) [upload=29636]: Waiting for error group managing segments...
Oct 31 21:22:07.951052914 (33h41m10.65264537s) [noise=true , address=145.79.227.32:4834 , piece_id=LBXTDM2QLBWFNDJ5272FY546TCRWC3IFYRNOBCCIS3KQDFMVJDFQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.951804679 (33h41m10.653397134s) [noise=true , address=145.79.227.15:4875 , piece_id=GXNV3O7D7URXRTLWVXMFMVTXGHZ4PAZMGQLI3L5SNUX3LDV7OXJA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.951868872 (33h41m10.653461324s) [noise=true , address=145.79.227.31:4798 , piece_id=QIYHHYEZGJEMENOVRJIDH7722MX2HI7XAGWUSFKIV3YNHKVPVD2Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952470935 (33h41m10.654063385s) [noise=true , address=145.79.227.30:4762 , piece_id=4BL5PJ3S2OC7PVSILA4IQCFT4JW6W5GBOXI7WRAS52DNBCF3DIHQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952500937 (33h41m10.654093387s) [noise=true , address=145.79.227.27:4689 , piece_id=E2674435DTU7SSPTAFGMRLBKZNPZAOIFMBDN5N734CP4RG23SVCA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952980719 (33h41m10.654573168s) [noise=true , address=145.79.227.28:4725 , piece_id=2UEZO2DIIDOVE4V6TXLKZAWXTOUNTIJKCBOP6JHREEEYBUWEXAUA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.953254355 (33h41m10.654846814s) [noise=true , address=145.79.227.11:4729 , piece_id=ZYPWL4SEIZHXYOAW4I5JOCPXDTHRCMO2FXYGA4ZUNHA2CSZ6DG5Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.953604050 (33h41m10.6551965s) [noise=true , address=145.79.227.15:4873 , piece_id=FIIZHJVUTZ42TATHLHZEWSRMVUFB7DYTWGIABNR4LJQYVH4SKQ5A, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.954075643 (33h41m10.6556681s) [noise=true , address=145.79.227.9:4367 , piece_id=Q2S6IK6OE4535GKC6J55MV3N45EUDWGOYKWS3ISQSZJKW2WJCD6Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.958770427 (33h41m10.660362881s) [noise=true , address=145.79.227.10:4401 , piece_id=YCZ6LL4ECE2QKCLEAF77L3C72VWCZ6IQVJLJ77RIACX4LIK2IJJA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.965022314 (33h41m10.666614767s) [noise=true , address=145.79.227.30:4763 , piece_id=ET72Q2LHTA3HQY2GQNCO2ZEOBP526EOWM72TL5XRQYOLLARFPZLQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.976650683 (33h41m10.678243138s) [noise=true , address=145.79.227.14:4840 , piece_id=MKR2JQOAFU6BA6JUG4LTAICYEDRI367JLXCNW6EPZWIJR2ZNKXYA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:36.799544661 (33h41m39.501137117s) [noise=true , address=45.91.116.4:4939 , piece_id=VYJO7TDD6SANMOZ7BB62KYLU72ACLUIZI7IHFV6I2PDAV7K7WLCQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection
Oct 31 21:22:36.799605331 (33h41m39.50119779s) [noise=true , address=77.90.31.12:4590 , piece_id=WQQC3535ERLWQOD6WCWJF3DUY4FY4CZF7KPD6Y2K3Q2NYQCP45JA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection; write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection
Oct 31 21:22:36.799747990 (33h41m39.501340451s) [noise=true , address=45.91.116.4:4940 , piece_id=7TW5URZLYHRH7MQX3PZGMYPKJS3SYTXCJJDSKCOQZHB2HIS6J4OA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 139.59.144.22:36232->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:36.799846589 (33h41m39.501439048s) [noise=true , address=45.91.116.8:4045 , piece_id=B6C3QMJBWPBBJBTSYFP67R5WOVVGIGOYN6B6P5BAW35FXYEVPW3A, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:39300->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:36.799901279 (33h41m39.501493715s) [seg_pos={0 0}, upload=29636]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:22:36.906137907 (33h41m39.607730362s) [upload=29636]: Done uploading segments...
Oct 31 21:22:36.906171689 (33h41m39.60776414s) [upload=29636]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 4.069234782s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [.....X....XX...................X..............X.........X...] duration:
2.118836861s trace:
https://jaeger.dev.storj.io/trace/5b5403da1e956ee2
FAIL (2025-10-31 21:23:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.118865217s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [................................X.........................XX] duration:
58.37472144s trace:
https://jaeger.dev.storj.io/trace/3173f13c3ae66d35
FAIL (2025-10-31 21:23:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
NOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.439072473 (33h41m54.140664943s) [noise=true , address=145.79.227.33:4888 , piece_id=55HIHYQGP4PYV47T3EOPUOOWH74Q2AC66ZJAVNMNY2NUDV5DM7QA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.440028262 (33h41m54.141620722s) [noise=true , address=145.79.227.36:4549 , piece_id=OPMGCVNCWD44POIDEO46A4ZKUJ4BQE7UDPSIF3UAIDWZP2QJRC4A, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.440879298 (33h41m54.142471755s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.441210726 (33h41m54.142803185s) [noise=true , address=77.90.31.19:4865 , piece_id=ZDUGAOKKTEBBUF4S7ZFOGSBITGLV5BUW2RHCVHVSDVPEYNTEWDOQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.441758919 (33h41m54.143351377s) [noise=true , address=77.90.31.12:4609 , piece_id=N4LUJ74AFB7JAE4ZL66HLKG746LGHKNFBY7TZTSTQOTIBPQN7I2Q, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442265638 (33h41m54.143858095s) [noise=true , address=45.91.116.10:4113 , piece_id=PIGRIKVRZCWODXJVZ4SIORYLZYWRMMB2EXQRZGJZRK3JCXLWRKOA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442303271 (33h41m54.14389573s) [noise=true , address=77.90.31.21:4910 , piece_id=K4ERYFVADFU242IHSKR6BGQJHLUNSQPIHNFHPA7QLNF7JSJSACZQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442656746 (33h41m54.144249202s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442782370 (33h41m54.144374829s) [noise=true , address=45.91.116.9:4286 , piece_id=3T3JM6QB6GN4OUSP2LN7ICHBZL3CO5GW56ODFBB7KVFNXSHRD52A, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443021968 (33h41m54.144614421s) [noise=true , address=45.91.116.8:4057 , piece_id=BVGJB2U2MHIEUOUOPF5GF6SY63DVJQ54LLI4W7SND3DZX56MZZUQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443252911 (33h41m54.144845368s) [noise=true , address=77.90.31.21:4911 , piece_id=FMIATBPBQP237AJC3IMMG7KC74M424BQ2KYSXJQOZHUMIIXXAVWQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443438074 (33h41m54.145030531s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443753610 (33h41m54.145346067s) [noise=true , address=145.79.227.6:4269 , piece_id=ZEEU26BQ4H4WTK4TMHGJYQPTQMQNQOWJGAECEINQCQDKTKATZ7DA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443851896 (33h41m54.145444348s) [noise=true , address=45.91.116.5:4308 , piece_id=FTWXK442NYHAARIK7REHW7GHHQ6K4URWJD75K6QNYXCI3S3BMKWA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.444315561 (33h41m54.145908018s) [noise=true , address=45.91.116.12:4242 , piece_id=TXGM7MH6RQ33AHDFWD4X3DSNZGL4DP4WBEFUFQXW5HEUPE56JWYA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.445026431 (33h41m54.14661889s) [noise=true , address=45.91.116.13:4177 , piece_id=XREU7IAJVWS2MJIEEEFOVKQOIAXJ3DGSX3NDA7KJUL6IUIXMX4PQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.445715106 (33h41m54.147307563s) [noise=true , address=145.79.227.52:4461 , piece_id=LQJVVKNJNZOVEXRW7Y5KGLX4IDFZMCG65UN3BAVPJEIHWN3NIWQA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:52.002260082 (33h41m54.703852532s) [upload=29647]: Next returned nil segment
Oct 31 21:22:52.002281375 (33h41m54.703873836s) [upload=29647]: Waiting for error group managing segments...
Oct 31 21:22:52.166910441 (33h41m54.868502889s) [noise=true , address=145.79.227.33:4888 , piece_id=55HIHYQGP4PYV47T3EOPUOOWH74Q2AC66ZJAVNMNY2NUDV5DM7QA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.171085961 (33h41m54.872678409s) [noise=true , address=145.79.227.36:4549 , piece_id=OPMGCVNCWD44POIDEO46A4ZKUJ4BQE7UDPSIF3UAIDWZP2QJRC4A, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.173190935 (33h41m54.874783383s) [noise=true , address=145.79.227.52:4461 , piece_id=LQJVVKNJNZOVEXRW7Y5KGLX4IDFZMCG65UN3BAVPJEIHWN3NIWQA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.174554840 (33h41m54.876147316s) [noise=true , address=145.79.227.6:4269 , piece_id=ZEEU26BQ4H4WTK4TMHGJYQPTQMQNQOWJGAECEINQCQDKTKATZ7DA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.870199575 (33h41m55.571792024s) [noise=true , address=45.91.116.8:4057 , piece_id=BVGJB2U2MHIEUOUOPF5GF6SY63DVJQ54LLI4W7SND3DZX56MZZUQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.932896280 (33h41m55.634488728s) [noise=true , address=45.91.116.5:4308 , piece_id=FTWXK442NYHAARIK7REHW7GHHQ6K4URWJD75K6QNYXCI3S3BMKWA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.996436348 (33h41m55.698028803s) [noise=true , address=45.91.116.13:4177 , piece_id=XREU7IAJVWS2MJIEEEFOVKQOIAXJ3DGSX3NDA7KJUL6IUIXMX4PQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:53.303664204 (33h41m56.005256643s) [noise=true , address=45.91.116.10:4113 , piece_id=PIGRIKVRZCWODXJVZ4SIORYLZYWRMMB2EXQRZGJZRK3JCXLWRKOA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:53.934904546 (33h41m56.636497004s) [noise=true , address=77.90.31.21:4910 , piece_id=K4ERYFVADFU242IHSKR6BGQJHLUNSQPIHNFHPA7QLNF7JSJSACZQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:54.224811350 (33h41m56.926403798s) [noise=true , address=77.90.31.12:4609 , piece_id=N4LUJ74AFB7JAE4ZL66HLKG746LGHKNFBY7TZTSTQOTIBPQN7I2Q, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:54.652552655 (33h41m57.35414511s) [noise=true , address=45.91.116.9:4286 , piece_id=3T3JM6QB6GN4OUSP2LN7ICHBZL3CO5GW56ODFBB7KVFNXSHRD52A, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:55.813839187 (33h41m58.515431634s) [noise=true , address=45.91.116.12:4242 , piece_id=TXGM7MH6RQ33AHDFWD4X3DSNZGL4DP4WBEFUFQXW5HEUPE56JWYA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:56.776728453 (33h41m59.4783209s) [noise=true , address=77.90.31.21:4911 , piece_id=FMIATBPBQP237AJC3IMMG7KC74M424BQ2KYSXJQOZHUMIIXXAVWQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:58.637065006 (33h42m1.338657444s) [noise=true , address=77.90.31.19:4865 , piece_id=ZDUGAOKKTEBBUF4S7ZFOGSBITGLV5BUW2RHCVHVSDVPEYNTEWDOQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:58.637090306 (33h42m1.338682758s) [seg_pos={0 0}, upload=29647]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:22:58.637316080 (33h42m1.338908539s) [noise=true , address=77.90.31.17:4787 , piece_id=V3EHNOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637326708 (33h42m1.338919172s) [noise=true , address=77.90.31.17:4787 , piece_id=V3EHNOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637380935 (33h42m1.338973393s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637394866 (33h42m1.338987321s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637446506 (33h42m1.339038963s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637456777 (33h42m1.339049231s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637522129 (33h42m1.339114607s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637528827 (33h42m1.339121275s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637546736 (33h42m1.339139173s) [seg_pos={0 0}, upload=29647]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:22:58.637649415 (33h42m1.339241873s) [upload=29647]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:22:58.637657672 (33h42m1.339250145s) [upload=29647]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:22:58.787848722 (33h42m1.489441173s) [upload=29647]: Done uploading segments...
Oct 31 21:22:58.787868799 (33h42m1.489461251s) [upload=29647]: Upload finished. err: <nil>
Duration: 58.374768339s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X....X..................X...............X...........X...] duration:
2.525392816s trace:
https://jaeger.dev.storj.io/trace/5f0044a50b568df8
FAIL (2025-10-31 21:23:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.525436439s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [......X....XX...................X..............X.........X..] duration:
1.714953566s trace:
https://jaeger.dev.storj.io/trace/5b5403da1e956ee2
FAIL (2025-10-31 21:23:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.714995893s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
8.355306945s trace:
https://jaeger.dev.storj.io/trace/62df97e1df2d7a
FAIL (2025-10-31 21:23:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:22:07.085611190 (33h41m9.787203641s) [upload=29636]: Starting upload
Oct 31 21:22:07.085688329 (33h41m9.787280767s) [upload=29636]: Done starting upload
Oct 31 21:22:07.085774737 (33h41m9.78736719s) [upload=29636]: Uploading segments...
Oct 31 21:22:07.086049239 (33h41m9.787641687s) [upload=29636]: Got next segment. Inline: false
Oct 31 21:22:07.086139291 (33h41m9.78773174s) [upload=29636]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:07.086145090 (33h41m9.787737538s) [upload=29636]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:07.198887636 (33h41m9.900480077s) [seg_pos={0 0}, upload=29636]: Begin upload segment...
Oct 31 21:22:07.198943347 (33h41m9.900535799s) [seg_pos={0 0}, upload=29636]: Done begin upload segment.
Oct 31 21:22:07.198976223 (33h41m9.900568674s) [noise=true , address=145.79.227.27:4689 , piece_id=E2674435DTU7SSPTAFGMRLBKZNPZAOIFMBDN5N734CP4RG23SVCA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199094043 (33h41m9.900686499s) [noise=true , address=145.79.227.15:4875 , piece_id=GXNV3O7D7URXRTLWVXMFMVTXGHZ4PAZMGQLI3L5SNUX3LDV7OXJA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199764319 (33h41m9.901356931s) [noise=true , address=145.79.227.14:4840 , piece_id=MKR2JQOAFU6BA6JUG4LTAICYEDRI367JLXCNW6EPZWIJR2ZNKXYA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199979140 (33h41m9.901571602s) [noise=true , address=145.79.227.28:4725 , piece_id=2UEZO2DIIDOVE4V6TXLKZAWXTOUNTIJKCBOP6JHREEEYBUWEXAUA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.201094736 (33h41m9.902687196s) [noise=true , address=145.79.227.10:4401 , piece_id=YCZ6LL4ECE2QKCLEAF77L3C72VWCZ6IQVJLJ77RIACX4LIK2IJJA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.201150898 (33h41m9.902743361s) [noise=true , address=145.79.227.32:4834 , piece_id=LBXTDM2QLBWFNDJ5272FY546TCRWC3IFYRNOBCCIS3KQDFMVJDFQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.202300922 (33h41m9.903893383s) [noise=true , address=45.91.116.4:4940 , piece_id=7TW5URZLYHRH7MQX3PZGMYPKJS3SYTXCJJDSKCOQZHB2HIS6J4OA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.202347708 (33h41m9.903940165s) [noise=true , address=77.90.31.12:4590 , piece_id=WQQC3535ERLWQOD6WCWJF3DUY4FY4CZF7KPD6Y2K3Q2NYQCP45JA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.203610241 (33h41m9.905202697s) [noise=true , address=145.79.227.31:4798 , piece_id=QIYHHYEZGJEMENOVRJIDH7722MX2HI7XAGWUSFKIV3YNHKVPVD2Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.204358358 (33h41m9.905950816s) [noise=true , address=145.79.227.9:4367 , piece_id=Q2S6IK6OE4535GKC6J55MV3N45EUDWGOYKWS3ISQSZJKW2WJCD6Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.204958129 (33h41m9.906550588s) [noise=true , address=45.91.116.4:4939 , piece_id=VYJO7TDD6SANMOZ7BB62KYLU72ACLUIZI7IHFV6I2PDAV7K7WLCQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.206076686 (33h41m9.907669138s) [noise=true , address=145.79.227.30:4763 , piece_id=ET72Q2LHTA3HQY2GQNCO2ZEOBP526EOWM72TL5XRQYOLLARFPZLQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.207450031 (33h41m9.909042495s) [noise=true , address=45.91.116.8:4045 , piece_id=B6C3QMJBWPBBJBTSYFP67R5WOVVGIGOYN6B6P5BAW35FXYEVPW3A, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.208460871 (33h41m9.910053312s) [noise=true , address=145.79.227.15:4873 , piece_id=FIIZHJVUTZ42TATHLHZEWSRMVUFB7DYTWGIABNR4LJQYVH4SKQ5A, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.209461830 (33h41m9.911054339s) [noise=true , address=145.79.227.11:4729 , piece_id=ZYPWL4SEIZHXYOAW4I5JOCPXDTHRCMO2FXYGA4ZUNHA2CSZ6DG5Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.210046720 (33h41m9.911639172s) [noise=true , address=145.79.227.30:4762 , piece_id=4BL5PJ3S2OC7PVSILA4IQCFT4JW6W5GBOXI7WRAS52DNBCF3DIHQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.808217579 (33h41m10.509810059s) [upload=29636]: Next returned nil segment
Oct 31 21:22:07.808236009 (33h41m10.509828464s) [upload=29636]: Waiting for error group managing segments...
Oct 31 21:22:07.951052914 (33h41m10.65264537s) [noise=true , address=145.79.227.32:4834 , piece_id=LBXTDM2QLBWFNDJ5272FY546TCRWC3IFYRNOBCCIS3KQDFMVJDFQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.951804679 (33h41m10.653397134s) [noise=true , address=145.79.227.15:4875 , piece_id=GXNV3O7D7URXRTLWVXMFMVTXGHZ4PAZMGQLI3L5SNUX3LDV7OXJA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.951868872 (33h41m10.653461324s) [noise=true , address=145.79.227.31:4798 , piece_id=QIYHHYEZGJEMENOVRJIDH7722MX2HI7XAGWUSFKIV3YNHKVPVD2Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952470935 (33h41m10.654063385s) [noise=true , address=145.79.227.30:4762 , piece_id=4BL5PJ3S2OC7PVSILA4IQCFT4JW6W5GBOXI7WRAS52DNBCF3DIHQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952500937 (33h41m10.654093387s) [noise=true , address=145.79.227.27:4689 , piece_id=E2674435DTU7SSPTAFGMRLBKZNPZAOIFMBDN5N734CP4RG23SVCA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952980719 (33h41m10.654573168s) [noise=true , address=145.79.227.28:4725 , piece_id=2UEZO2DIIDOVE4V6TXLKZAWXTOUNTIJKCBOP6JHREEEYBUWEXAUA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.953254355 (33h41m10.654846814s) [noise=true , address=145.79.227.11:4729 , piece_id=ZYPWL4SEIZHXYOAW4I5JOCPXDTHRCMO2FXYGA4ZUNHA2CSZ6DG5Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.953604050 (33h41m10.6551965s) [noise=true , address=145.79.227.15:4873 , piece_id=FIIZHJVUTZ42TATHLHZEWSRMVUFB7DYTWGIABNR4LJQYVH4SKQ5A, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.954075643 (33h41m10.6556681s) [noise=true , address=145.79.227.9:4367 , piece_id=Q2S6IK6OE4535GKC6J55MV3N45EUDWGOYKWS3ISQSZJKW2WJCD6Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.958770427 (33h41m10.660362881s) [noise=true , address=145.79.227.10:4401 , piece_id=YCZ6LL4ECE2QKCLEAF77L3C72VWCZ6IQVJLJ77RIACX4LIK2IJJA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.965022314 (33h41m10.666614767s) [noise=true , address=145.79.227.30:4763 , piece_id=ET72Q2LHTA3HQY2GQNCO2ZEOBP526EOWM72TL5XRQYOLLARFPZLQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.976650683 (33h41m10.678243138s) [noise=true , address=145.79.227.14:4840 , piece_id=MKR2JQOAFU6BA6JUG4LTAICYEDRI367JLXCNW6EPZWIJR2ZNKXYA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:36.799544661 (33h41m39.501137117s) [noise=true , address=45.91.116.4:4939 , piece_id=VYJO7TDD6SANMOZ7BB62KYLU72ACLUIZI7IHFV6I2PDAV7K7WLCQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection
Oct 31 21:22:36.799605331 (33h41m39.50119779s) [noise=true , address=77.90.31.12:4590 , piece_id=WQQC3535ERLWQOD6WCWJF3DUY4FY4CZF7KPD6Y2K3Q2NYQCP45JA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection; write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection
Oct 31 21:22:36.799747990 (33h41m39.501340451s) [noise=true , address=45.91.116.4:4940 , piece_id=7TW5URZLYHRH7MQX3PZGMYPKJS3SYTXCJJDSKCOQZHB2HIS6J4OA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 139.59.144.22:36232->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:36.799846589 (33h41m39.501439048s) [noise=true , address=45.91.116.8:4045 , piece_id=B6C3QMJBWPBBJBTSYFP67R5WOVVGIGOYN6B6P5BAW35FXYEVPW3A, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:39300->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:36.799901279 (33h41m39.501493715s) [seg_pos={0 0}, upload=29636]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:22:36.906137907 (33h41m39.607730362s) [upload=29636]: Done uploading segments...
Oct 31 21:22:36.906171689 (33h41m39.60776414s) [upload=29636]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 8.355336486s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X....X..................X...............X...........X..] duration:
3.674840171s trace:
https://jaeger.dev.storj.io/trace/5f0044a50b568df8
FAIL (2025-10-31 21:23:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.6748696s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X....X..................X...............X...........X.] duration:
3.275992596s trace:
https://jaeger.dev.storj.io/trace/5f0044a50b568df8
FAIL (2025-10-31 21:22:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.276015887s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X.........................X] duration:
1m0.105613233s trace:
https://jaeger.dev.storj.io/trace/3cdbdd8761f05348
FAIL (2025-10-31 21:22:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
MDDQYOYQBXE5F3MIJHGA, seg_pos={0 0}, upload=29630]: Uploading piece...
Oct 31 21:21:37.973662376 (33h40m40.675254827s) [noise=true , address=77.90.31.18:4816 , piece_id=T7HGEFNWIKV7C7JM2PLBLG5ZILNGTYGQSXSI6LH2QEGO3KQT6NRA, seg_pos={0 0}, upload=29630]: Uploading piece...
Oct 31 21:21:37.975091095 (33h40m40.676683552s) [noise=true , address=145.79.227.40:4060 , piece_id=CEHI5D4WHRDTPDBFPIV2UBINOFWGJKYJH4DGLDVDVZPBPE4GZMPQ, seg_pos={0 0}, upload=29630]: Uploading piece...
Oct 31 21:21:37.977677907 (33h40m40.679270364s) [noise=true , address=77.90.31.21:4883 , piece_id=VR4ZBDXPLBKJCWUWNE3AWO6RXBKXG2QJHQQD4CU2OVRZ6F4MLHUQ, seg_pos={0 0}, upload=29630]: Uploading piece...
Oct 31 21:21:37.983110230 (33h40m40.684702747s) [noise=true , address=77.90.31.11:4564 , piece_id=OOHV3AT5FQLXJKW5UGDKY2C4CIBPEBHP52RU4XOOUXN3CZT54JAA, seg_pos={0 0}, upload=29630]: Uploading piece...
Oct 31 21:21:37.983499171 (33h40m40.685091628s) [noise=true , address=45.91.116.12:4253 , piece_id=KRLW233IKPVQXIOZVPURCC6AV7WIPGVS4HUAUUM4D4TH3OIPY6UA, seg_pos={0 0}, upload=29630]: Uploading piece...
Oct 31 21:21:37.985071008 (33h40m40.686663461s) [noise=true , address=145.79.227.7:4317 , piece_id=NKZKIQ7PZXXKBMPFJMHKQ33Y2VACIHKEJFBTTKAUQDE2U43HZNHA, seg_pos={0 0}, upload=29630]: Uploading piece...
Oct 31 21:21:37.988103155 (33h40m40.68969561s) [noise=true , address=77.90.31.15:4710 , piece_id=FYUXBY6UDRRK274TOH67BADV5GUPGI6EM2O7DGQ3363W7XNAEUBA, seg_pos={0 0}, upload=29630]: Uploading piece...
Oct 31 21:21:37.991831428 (33h40m40.693423887s) [noise=true , address=145.79.227.39:4032 , piece_id=Q6QAHPL3GMYFCUD2BMUJLFUKQKZVS3VDAVSUC7GRUMC2PLM2UCHA, seg_pos={0 0}, upload=29630]: Uploading piece...
Oct 31 21:21:37.995305068 (33h40m40.696897506s) [noise=true , address=45.91.116.7:4028 , piece_id=FZUP2W7LXUR67IRMRUPJ5XC7O3LZPLDPAKDLXQZPYGJ7XPCPYT5Q, seg_pos={0 0}, upload=29630]: Uploading piece...
Oct 31 21:21:37.998714908 (33h40m40.700307365s) [noise=true , address=77.90.31.16:4768 , piece_id=FADT46KM37RX3O4GGVJ5U5FB7C4SASDXULMJVWAPWWCDPNEXUXAA, seg_pos={0 0}, upload=29630]: Uploading piece...
Oct 31 21:21:38.000747429 (33h40m40.702339906s) [noise=true , address=77.90.31.14:4675 , piece_id=CVBTHDYA47JNY22ATPVSGWTRPLLFZ6MZQEL6AK2MXTROVQR5FNRA, seg_pos={0 0}, upload=29630]: Uploading piece...
Oct 31 21:21:38.537203780 (33h40m41.238796218s) [upload=29630]: Next returned nil segment
Oct 31 21:21:38.537217876 (33h40m41.238810315s) [upload=29630]: Waiting for error group managing segments...
Oct 31 21:21:38.708235212 (33h40m41.409827661s) [noise=true , address=145.79.227.5:4225 , piece_id=JLSAJG6XUXL7ACMYFSXYUAXDZCPPUWXU45S2OQ63C2RHELPV5HGQ, seg_pos={0 0}, upload=29630]: Done uploading piece. err: <nil>
Oct 31 21:21:38.717459013 (33h40m41.419051469s) [noise=true , address=145.79.227.9:4381 , piece_id=QDMVUE3JHRKKIJN5HBHYEXKSP6YLCFHBKPXJ5FR7XU6PM6FFL5FA, seg_pos={0 0}, upload=29630]: Done uploading piece. err: <nil>
Oct 31 21:21:38.719218277 (33h40m41.420810736s) [noise=true , address=145.79.227.52:4476 , piece_id=D5JFZMJHYEXDEICUKXWAP3INVU7DE5MGMDDQYOYQBXE5F3MIJHGA, seg_pos={0 0}, upload=29630]: Done uploading piece. err: <nil>
Oct 31 21:21:38.723115417 (33h40m41.424707866s) [noise=true , address=145.79.227.40:4060 , piece_id=CEHI5D4WHRDTPDBFPIV2UBINOFWGJKYJH4DGLDVDVZPBPE4GZMPQ, seg_pos={0 0}, upload=29630]: Done uploading piece. err: <nil>
Oct 31 21:21:38.736338925 (33h40m41.437931363s) [noise=true , address=145.79.227.39:4032 , piece_id=Q6QAHPL3GMYFCUD2BMUJLFUKQKZVS3VDAVSUC7GRUMC2PLM2UCHA, seg_pos={0 0}, upload=29630]: Done uploading piece. err: <nil>
Oct 31 21:21:38.740700760 (33h40m41.442293209s) [noise=true , address=145.79.227.7:4317 , piece_id=NKZKIQ7PZXXKBMPFJMHKQ33Y2VACIHKEJFBTTKAUQDE2U43HZNHA, seg_pos={0 0}, upload=29630]: Done uploading piece. err: <nil>
Oct 31 21:22:34.722822633 (33h41m37.424415082s) [noise=true , address=77.90.31.16:4768 , piece_id=FADT46KM37RX3O4GGVJ5U5FB7C4SASDXULMJVWAPWWCDPNEXUXAA, seg_pos={0 0}, upload=29630]: Done uploading piece. err: <nil>
Oct 31 21:22:37.569895309 (33h41m40.271487763s) [noise=true , address=77.90.31.10:4539 , piece_id=WAMQIL7X64JQEZYMKMMI6BMGI2GU6EEKEHOKNTB7MSYMLQKJJ3CA, seg_pos={0 0}, upload=29630]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 139.59.144.22:58986->77.90.31.10:4539: use of closed network connection; write tcp 139.59.144.22:58986->77.90.31.10:4539: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58986->77.90.31.10:4539: use of closed network connection
Oct 31 21:22:37.570095394 (33h41m40.27168785s) [noise=true , address=77.90.31.21:4883 , piece_id=VR4ZBDXPLBKJCWUWNE3AWO6RXBKXG2QJHQQD4CU2OVRZ6F4MLHUQ, seg_pos={0 0}, upload=29630]: Done uploading piece. err: ecclient: upload failed (node:191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, address:77.90.31.21:4883): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:22:37.570296522 (33h41m40.27188897s) [noise=true , address=77.90.31.15:4710 , piece_id=FYUXBY6UDRRK274TOH67BADV5GUPGI6EM2O7DGQ3363W7XNAEUBA, seg_pos={0 0}, upload=29630]: Done uploading piece. err: ecclient: upload failed (node:12JLoKRZfmGsqVbKxUNR9XbSjh7yauChgja9hkbNxukYSuj1HJC, address:77.90.31.15:4710): protocol: read tcp 139.59.144.22:57240->77.90.31.15:4710: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:37.570333986 (33h41m40.271926445s) [noise=true , address=77.90.31.11:4564 , piece_id=OOHV3AT5FQLXJKW5UGDKY2C4CIBPEBHP52RU4XOOUXN3CZT54JAA, seg_pos={0 0}, upload=29630]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 139.59.144.22:48100->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:48100->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48100->77.90.31.11:4564: use of closed network connection
Oct 31 21:22:37.570625336 (33h41m40.272217794s) [noise=true , address=77.90.31.14:4675 , piece_id=CVBTHDYA47JNY22ATPVSGWTRPLLFZ6MZQEL6AK2MXTROVQR5FNRA, seg_pos={0 0}, upload=29630]: Done uploading piece. err: ecclient: upload failed (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k, address:77.90.31.14:4675): protocol: write tcp 139.59.144.22:39520->77.90.31.14:4675: use of closed network connection; write tcp 139.59.144.22:39520->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39520->77.90.31.14:4675: use of closed network connection
Oct 31 21:22:37.570679844 (33h41m40.272272295s) [noise=true , address=45.91.116.12:4253 , piece_id=KRLW233IKPVQXIOZVPURCC6AV7WIPGVS4HUAUUM4D4TH3OIPY6UA, seg_pos={0 0}, upload=29630]: Done uploading piece. err: ecclient: upload failed (node:12gDvXYhShpWvptTU8bivGGwYBTyMF3PJcRv6RJ66vm8aZewejn, address:45.91.116.12:4253): protocol: read tcp 139.59.144.22:55692->45.91.116.12:4253: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:37.570910782 (33h41m40.272503231s) [noise=true , address=77.90.31.18:4816 , piece_id=T7HGEFNWIKV7C7JM2PLBLG5ZILNGTYGQSXSI6LH2QEGO3KQT6NRA, seg_pos={0 0}, upload=29630]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 139.59.144.22:33134->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:37.570981227 (33h41m40.272573681s) [noise=true , address=45.91.116.8:4058 , piece_id=NO2PGYT33H5G3CYQQIUCRVM5WW45VZ5XHQ5NW4IKJUW6OWGKORAQ, seg_pos={0 0}, upload=29630]: Done uploading piece. err: ecclient: upload failed (node:12GHuEEAGNMcY8EM4n2kFnvtA7p46ZzNHSZYw7yaXdTRCU1Kq6B, address:45.91.116.8:4058): protocol: read tcp 139.59.144.22:35464->45.91.116.8:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:37.571078678 (33h41m40.272671126s) [noise=true , address=45.91.116.7:4028 , piece_id=FZUP2W7LXUR67IRMRUPJ5XC7O3LZPLDPAKDLXQZPYGJ7XPCPYT5Q, seg_pos={0 0}, upload=29630]: Done uploading piece. err: ecclient: upload failed (node:1g3sbXJir88dVbuJCShqFZ5KH2RJhq1kSqfVx6wN7ZkQtucnrB, address:45.91.116.7:4028): protocol: read tcp 139.59.144.22:47986->45.91.116.7:4028: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:37.571261619 (33h41m40.272854068s) [noise=true , address=45.91.116.14:4210 , piece_id=73ALT6COTTZ4IXFP4LW3HSEQXQRAHYA65JS7F73QAXS6NLYJQC3A, seg_pos={0 0}, upload=29630]: Done uploading piece. err: ecclient: upload failed (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n, address:45.91.116.14:4210): protocol: read tcp 139.59.144.22:58222->45.91.116.14:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:37.571394296 (33h41m40.272986744s) [noise=true , address=45.91.116.4:4943 , piece_id=YSG3XVUGKOGEYF7J325AYWFNZFZ6NMSVU2MY4JHHUYFCG3SIWLKQ, seg_pos={0 0}, upload=29630]: Done uploading piece. err: ecclient: upload failed (node:128QuaUBgwagbp1fCbMDLMncgCth8bd4kaMTnd4ebP1pvQMB1jv, address:45.91.116.4:4943): protocol: read tcp 139.59.144.22:41558->45.91.116.4:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:37.571428753 (33h41m40.273021195s) [seg_pos={0 0}, upload=29630]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Oct 31 21:22:37.672847208 (33h41m40.374439662s) [upload=29630]: Done uploading segments...
Oct 31 21:22:37.672904620 (33h41m40.374497076s) [upload=29630]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 1m0.105636155s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.108875028s trace:
https://jaeger.dev.storj.io/trace/62df97e1df2d7a
FAIL (2025-10-31 21:22:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:22:07.085611190 (33h41m9.787203641s) [upload=29636]: Starting upload
Oct 31 21:22:07.085688329 (33h41m9.787280767s) [upload=29636]: Done starting upload
Oct 31 21:22:07.085774737 (33h41m9.78736719s) [upload=29636]: Uploading segments...
Oct 31 21:22:07.086049239 (33h41m9.787641687s) [upload=29636]: Got next segment. Inline: false
Oct 31 21:22:07.086139291 (33h41m9.78773174s) [upload=29636]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:07.086145090 (33h41m9.787737538s) [upload=29636]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:07.198887636 (33h41m9.900480077s) [seg_pos={0 0}, upload=29636]: Begin upload segment...
Oct 31 21:22:07.198943347 (33h41m9.900535799s) [seg_pos={0 0}, upload=29636]: Done begin upload segment.
Oct 31 21:22:07.198976223 (33h41m9.900568674s) [noise=true , address=145.79.227.27:4689 , piece_id=E2674435DTU7SSPTAFGMRLBKZNPZAOIFMBDN5N734CP4RG23SVCA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199094043 (33h41m9.900686499s) [noise=true , address=145.79.227.15:4875 , piece_id=GXNV3O7D7URXRTLWVXMFMVTXGHZ4PAZMGQLI3L5SNUX3LDV7OXJA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199764319 (33h41m9.901356931s) [noise=true , address=145.79.227.14:4840 , piece_id=MKR2JQOAFU6BA6JUG4LTAICYEDRI367JLXCNW6EPZWIJR2ZNKXYA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199979140 (33h41m9.901571602s) [noise=true , address=145.79.227.28:4725 , piece_id=2UEZO2DIIDOVE4V6TXLKZAWXTOUNTIJKCBOP6JHREEEYBUWEXAUA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.201094736 (33h41m9.902687196s) [noise=true , address=145.79.227.10:4401 , piece_id=YCZ6LL4ECE2QKCLEAF77L3C72VWCZ6IQVJLJ77RIACX4LIK2IJJA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.201150898 (33h41m9.902743361s) [noise=true , address=145.79.227.32:4834 , piece_id=LBXTDM2QLBWFNDJ5272FY546TCRWC3IFYRNOBCCIS3KQDFMVJDFQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.202300922 (33h41m9.903893383s) [noise=true , address=45.91.116.4:4940 , piece_id=7TW5URZLYHRH7MQX3PZGMYPKJS3SYTXCJJDSKCOQZHB2HIS6J4OA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.202347708 (33h41m9.903940165s) [noise=true , address=77.90.31.12:4590 , piece_id=WQQC3535ERLWQOD6WCWJF3DUY4FY4CZF7KPD6Y2K3Q2NYQCP45JA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.203610241 (33h41m9.905202697s) [noise=true , address=145.79.227.31:4798 , piece_id=QIYHHYEZGJEMENOVRJIDH7722MX2HI7XAGWUSFKIV3YNHKVPVD2Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.204358358 (33h41m9.905950816s) [noise=true , address=145.79.227.9:4367 , piece_id=Q2S6IK6OE4535GKC6J55MV3N45EUDWGOYKWS3ISQSZJKW2WJCD6Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.204958129 (33h41m9.906550588s) [noise=true , address=45.91.116.4:4939 , piece_id=VYJO7TDD6SANMOZ7BB62KYLU72ACLUIZI7IHFV6I2PDAV7K7WLCQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.206076686 (33h41m9.907669138s) [noise=true , address=145.79.227.30:4763 , piece_id=ET72Q2LHTA3HQY2GQNCO2ZEOBP526EOWM72TL5XRQYOLLARFPZLQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.207450031 (33h41m9.909042495s) [noise=true , address=45.91.116.8:4045 , piece_id=B6C3QMJBWPBBJBTSYFP67R5WOVVGIGOYN6B6P5BAW35FXYEVPW3A, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.208460871 (33h41m9.910053312s) [noise=true , address=145.79.227.15:4873 , piece_id=FIIZHJVUTZ42TATHLHZEWSRMVUFB7DYTWGIABNR4LJQYVH4SKQ5A, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.209461830 (33h41m9.911054339s) [noise=true , address=145.79.227.11:4729 , piece_id=ZYPWL4SEIZHXYOAW4I5JOCPXDTHRCMO2FXYGA4ZUNHA2CSZ6DG5Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.210046720 (33h41m9.911639172s) [noise=true , address=145.79.227.30:4762 , piece_id=4BL5PJ3S2OC7PVSILA4IQCFT4JW6W5GBOXI7WRAS52DNBCF3DIHQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.808217579 (33h41m10.509810059s) [upload=29636]: Next returned nil segment
Oct 31 21:22:07.808236009 (33h41m10.509828464s) [upload=29636]: Waiting for error group managing segments...
Oct 31 21:22:07.951052914 (33h41m10.65264537s) [noise=true , address=145.79.227.32:4834 , piece_id=LBXTDM2QLBWFNDJ5272FY546TCRWC3IFYRNOBCCIS3KQDFMVJDFQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.951804679 (33h41m10.653397134s) [noise=true , address=145.79.227.15:4875 , piece_id=GXNV3O7D7URXRTLWVXMFMVTXGHZ4PAZMGQLI3L5SNUX3LDV7OXJA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.951868872 (33h41m10.653461324s) [noise=true , address=145.79.227.31:4798 , piece_id=QIYHHYEZGJEMENOVRJIDH7722MX2HI7XAGWUSFKIV3YNHKVPVD2Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952470935 (33h41m10.654063385s) [noise=true , address=145.79.227.30:4762 , piece_id=4BL5PJ3S2OC7PVSILA4IQCFT4JW6W5GBOXI7WRAS52DNBCF3DIHQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952500937 (33h41m10.654093387s) [noise=true , address=145.79.227.27:4689 , piece_id=E2674435DTU7SSPTAFGMRLBKZNPZAOIFMBDN5N734CP4RG23SVCA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952980719 (33h41m10.654573168s) [noise=true , address=145.79.227.28:4725 , piece_id=2UEZO2DIIDOVE4V6TXLKZAWXTOUNTIJKCBOP6JHREEEYBUWEXAUA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.953254355 (33h41m10.654846814s) [noise=true , address=145.79.227.11:4729 , piece_id=ZYPWL4SEIZHXYOAW4I5JOCPXDTHRCMO2FXYGA4ZUNHA2CSZ6DG5Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.953604050 (33h41m10.6551965s) [noise=true , address=145.79.227.15:4873 , piece_id=FIIZHJVUTZ42TATHLHZEWSRMVUFB7DYTWGIABNR4LJQYVH4SKQ5A, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.954075643 (33h41m10.6556681s) [noise=true , address=145.79.227.9:4367 , piece_id=Q2S6IK6OE4535GKC6J55MV3N45EUDWGOYKWS3ISQSZJKW2WJCD6Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.958770427 (33h41m10.660362881s) [noise=true , address=145.79.227.10:4401 , piece_id=YCZ6LL4ECE2QKCLEAF77L3C72VWCZ6IQVJLJ77RIACX4LIK2IJJA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.965022314 (33h41m10.666614767s) [noise=true , address=145.79.227.30:4763 , piece_id=ET72Q2LHTA3HQY2GQNCO2ZEOBP526EOWM72TL5XRQYOLLARFPZLQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.976650683 (33h41m10.678243138s) [noise=true , address=145.79.227.14:4840 , piece_id=MKR2JQOAFU6BA6JUG4LTAICYEDRI367JLXCNW6EPZWIJR2ZNKXYA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:36.799544661 (33h41m39.501137117s) [noise=true , address=45.91.116.4:4939 , piece_id=VYJO7TDD6SANMOZ7BB62KYLU72ACLUIZI7IHFV6I2PDAV7K7WLCQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection
Oct 31 21:22:36.799605331 (33h41m39.50119779s) [noise=true , address=77.90.31.12:4590 , piece_id=WQQC3535ERLWQOD6WCWJF3DUY4FY4CZF7KPD6Y2K3Q2NYQCP45JA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection; write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection
Oct 31 21:22:36.799747990 (33h41m39.501340451s) [noise=true , address=45.91.116.4:4940 , piece_id=7TW5URZLYHRH7MQX3PZGMYPKJS3SYTXCJJDSKCOQZHB2HIS6J4OA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 139.59.144.22:36232->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:36.799846589 (33h41m39.501439048s) [noise=true , address=45.91.116.8:4045 , piece_id=B6C3QMJBWPBBJBTSYFP67R5WOVVGIGOYN6B6P5BAW35FXYEVPW3A, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:39300->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:36.799901279 (33h41m39.501493715s) [seg_pos={0 0}, upload=29636]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:22:36.906137907 (33h41m39.607730362s) [upload=29636]: Done uploading segments...
Oct 31 21:22:36.906171689 (33h41m39.60776414s) [upload=29636]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.108900015s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [.......X....XX...................X..............X.........X.] duration:
2.750823719s trace:
https://jaeger.dev.storj.io/trace/5b5403da1e956ee2
FAIL (2025-10-31 21:22:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.750848921s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X....X..................X...............X...........X] duration:
30.000821655s trace:
https://jaeger.dev.storj.io/trace/5f0044a50b568df8
FAIL (2025-10-31 21:22:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000846372s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [........X....XX...................X..............X.........X] duration:
5.002864902s trace:
https://jaeger.dev.storj.io/trace/5b5403da1e956ee2
FAIL (2025-10-31 21:21:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002895462s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.........X....XX...................X..............X.........] duration:
1.613841418s trace:
https://jaeger.dev.storj.io/trace/135da41e6fd891e
FAIL (2025-10-31 21:21:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.613919056s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
2.280960436s trace:
https://jaeger.dev.storj.io/trace/3190c37ec1de204b
FAIL (2025-10-31 21:21:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
ece...
Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment
Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments...
Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection
Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection
Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection
Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection
Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection
Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection
Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection
Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments...
Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 2.280988092s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X....X..................X...............X...........] duration:
2.868422836s trace:
https://jaeger.dev.storj.io/trace/51e763c574611f36
FAIL (2025-10-31 21:21:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.868449948s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
2.336953214s trace:
https://jaeger.dev.storj.io/trace/3190c37ec1de204b
FAIL (2025-10-31 21:20:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
ece...
Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment
Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments...
Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection
Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection
Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection
Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection
Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection
Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection
Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection
Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments...
Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 2.3370458s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
2.206211797s trace:
https://jaeger.dev.storj.io/trace/3190c37ec1de204b
FAIL (2025-10-31 21:20:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
ece...
Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment
Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments...
Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection
Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection
Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection
Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection
Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection
Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection
Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection
Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments...
Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 2.206238008s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..........X....XX...................X..............X........] duration:
1.716591972s trace:
https://jaeger.dev.storj.io/trace/135da41e6fd891e
FAIL (2025-10-31 21:20:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.716616909s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........X....X..................X...............X..........] duration:
2.809285338s trace:
https://jaeger.dev.storj.io/trace/51e763c574611f36
FAIL (2025-10-31 21:20:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.809312075s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
2.427109872s trace:
https://jaeger.dev.storj.io/trace/3190c37ec1de204b
FAIL (2025-10-31 21:19:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
ece...
Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment
Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments...
Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection
Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection
Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection
Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection
Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection
Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection
Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection
Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments...
Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 2.427133641s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...........X....XX...................X..............X.......] duration:
1.646908429s trace:
https://jaeger.dev.storj.io/trace/135da41e6fd891e
FAIL (2025-10-31 21:19:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.64696005s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........X....X..................X...............X.........] duration:
2.666469787s trace:
https://jaeger.dev.storj.io/trace/51e763c574611f36
FAIL (2025-10-31 21:19:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.666502871s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
2.354140528s trace:
https://jaeger.dev.storj.io/trace/3190c37ec1de204b
FAIL (2025-10-31 21:19:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
ece...
Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment
Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments...
Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection
Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection
Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection
Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection
Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection
Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection
Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection
Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments...
Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 2.354165861s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........X....X..................X...............X........] duration:
2.674346074s trace:
https://jaeger.dev.storj.io/trace/51e763c574611f36
FAIL (2025-10-31 21:18:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.674466834s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [............X....XX...................X..............X......] duration:
1.624026428s trace:
https://jaeger.dev.storj.io/trace/135da41e6fd891e
FAIL (2025-10-31 21:18:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.624053599s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............X....X..................X...............X.......] duration:
2.558564657s trace:
https://jaeger.dev.storj.io/trace/51e763c574611f36
FAIL (2025-10-31 21:18:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.558590922s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
2.385255922s trace:
https://jaeger.dev.storj.io/trace/3190c37ec1de204b
FAIL (2025-10-31 21:17:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
ece...
Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment
Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments...
Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection
Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection
Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection
Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection
Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection
Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection
Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection
Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments...
Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 2.385281372s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............X....X..................X...............X......] duration:
2.613461536s trace:
https://jaeger.dev.storj.io/trace/51e763c574611f36
FAIL (2025-10-31 21:17:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.613496673s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.............X....XX...................X..............X.....] duration:
1.987505482s trace:
https://jaeger.dev.storj.io/trace/135da41e6fd891e
FAIL (2025-10-31 21:17:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.987526646s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
2.365324914s trace:
https://jaeger.dev.storj.io/trace/3190c37ec1de204b
FAIL (2025-10-31 21:17:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
ece...
Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment
Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments...
Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection
Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection
Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection
Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection
Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection
Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection
Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection
Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments...
Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 2.365353647s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............X....X..................X...............X.....] duration:
2.746703752s trace:
https://jaeger.dev.storj.io/trace/51e763c574611f36
FAIL (2025-10-31 21:17:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.746813477s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..............X....XX...................X..............X....] duration:
2.117060194s trace:
https://jaeger.dev.storj.io/trace/135da41e6fd891e
FAIL (2025-10-31 21:16:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.117086712s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
2.320283432s trace:
https://jaeger.dev.storj.io/trace/3190c37ec1de204b
FAIL (2025-10-31 21:16:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
ece...
Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment
Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments...
Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection
Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection
Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection
Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection
Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection
Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection
Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection
Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments...
Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 2.32030999s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............X....X..................X...............X....] duration:
2.526917231s trace:
https://jaeger.dev.storj.io/trace/51e763c574611f36
FAIL (2025-10-31 21:16:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.526942154s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...............X....XX...................X..............X...] duration:
1.713044016s trace:
https://jaeger.dev.storj.io/trace/135da41e6fd891e
FAIL (2025-10-31 21:16:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.713066053s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
2.353055952s trace:
https://jaeger.dev.storj.io/trace/3190c37ec1de204b
FAIL (2025-10-31 21:16:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
ece...
Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment
Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments...
Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection
Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection
Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection
Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection
Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection
Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection
Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection
Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments...
Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 2.353085948s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [................X....X..................X...............X...] duration:
2.645738432s trace:
https://jaeger.dev.storj.io/trace/51e763c574611f36
FAIL (2025-10-31 21:15:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.645820497s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.336221401s trace:
https://jaeger.dev.storj.io/trace/3190c37ec1de204b
FAIL (2025-10-31 21:15:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
ece...
Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment
Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments...
Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection
Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection
Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection
Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection
Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection
Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection
Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection
Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments...
Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 2.336315818s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................X....X..................X...............X..] duration:
2.831432088s trace:
https://jaeger.dev.storj.io/trace/51e763c574611f36
FAIL (2025-10-31 21:15:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.831502164s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [................X....XX...................X..............X..] duration:
1.688012854s trace:
https://jaeger.dev.storj.io/trace/135da41e6fd891e
FAIL (2025-10-31 21:15:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.688081076s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.................X....XX...................X..............X.] duration:
1.631579055s trace:
https://jaeger.dev.storj.io/trace/135da41e6fd891e
FAIL (2025-10-31 21:14:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.631600627s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
11.636034627s trace:
https://jaeger.dev.storj.io/trace/3190c37ec1de204b
FAIL (2025-10-31 21:14:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
ece...
Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece...
Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment
Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments...
Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil>
Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection
Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection
Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection
Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection
Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection
Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection
Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection
Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments...
Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 11.636059964s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................X....X..................X...............X.] duration:
2.867186228s trace:
https://jaeger.dev.storj.io/trace/51e763c574611f36
FAIL (2025-10-31 21:14:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.867212756s
edge
gateway
us-select-1
|