Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
10.577204085s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:15:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.577235392s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................XX...........................] duration:
3.037293142s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:15:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.037317584s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [..............................XX.......................XXXXX] duration:
30.000622944s trace:
https://jaeger.dev.storj.io/trace/5083ef3fcc9a0dfc
FAIL (2025-10-31 22:15:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000642881s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
2.946689101s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:14:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.946728989s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................XX..........................] duration:
3.001890545s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:14:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.00191852s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
10.541808369s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:14:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.541832641s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................................X.........................] duration:
3.11299847s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:14:17 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.113056104s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...............................XX.......................XXXX] duration:
30.000764071s trace:
https://jaeger.dev.storj.io/trace/771343e613ab3283
FAIL (2025-10-31 22:14:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000791819s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
2.933579008s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:14:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.933599533s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
14.934112276s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:13:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.934128363s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................XX.........................] duration:
3.023171547s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:13:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.023187437s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [X...............................X...........................] duration:
2.828202712s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:13:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.828226967s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [................................XX.......................XXX] duration:
30.001965214s trace:
https://jaeger.dev.storj.io/trace/37a5bef0ad7d4482
FAIL (2025-10-31 22:13:26 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00198461s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [XX...............................X..........................] duration:
2.884210859s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:13:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.884275292s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................................X........................] duration:
4.186568628s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:13:10 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.186596468s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................XX........................] duration:
3.175911886s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:13:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.175935279s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................................XX.......................XX] duration:
30.0010966s trace:
https://jaeger.dev.storj.io/trace/6818ef78319e8c13
FAIL (2025-10-31 22:12:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001121523s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [X...................................X.......................] duration:
2.146499545s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:12:26 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.146522144s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
5.685934303s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:12:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.686045914s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X..................................XX.......................] duration:
3.001343704s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:12:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.001381401s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.XX...............................X.........................] duration:
2.351786612s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:12:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.351828987s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..XX...............................X........................] duration:
2.789313159s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:11:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.789336976s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.X...................................X......................] duration:
1.98863889s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:11:57 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.988724107s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................XX.......................X] duration:
30.001309111s trace:
https://jaeger.dev.storj.io/trace/3f374088a5e6c7a1
FAIL (2025-10-31 22:11:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001331463s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X..................................XX......................] duration:
3.055539588s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:11:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.055564202s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [X.................................X.........................] duration:
3.496751624s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:11:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.49678315s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..X...................................X.....................] duration:
2.884181953s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:11:26 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.884224751s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...XX...............................X.......................] duration:
2.877763446s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:11:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.87779299s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X..................................XX.......................] duration:
2.903318377s trace:
https://jaeger.dev.storj.io/trace/1cc17f34cdfb5991
FAIL (2025-10-31 22:11:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.903358938s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.X.................................X........................] duration:
2.889536387s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:10:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.889566202s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...X...................................X....................] duration:
2.208789277s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:10:51 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.208832024s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X.X..................................XX.....................] duration:
2.384756654s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:10:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.384783964s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [XX..................................XX......................] duration:
2.723948339s trace:
https://jaeger.dev.storj.io/trace/1cc17f34cdfb5991
FAIL (2025-10-31 22:10:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.724042488s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [....XX...............................X......................] duration:
2.343690271s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:10:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.343709045s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....XX...............................X.....................] duration:
2.42700244s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:10:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.42705203s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X.X..................................XX....................] duration:
2.280884167s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:09:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.280930223s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.XX..................................XX.....................] duration:
2.680691339s trace:
https://jaeger.dev.storj.io/trace/1cc17f34cdfb5991
FAIL (2025-10-31 22:09:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.680711418s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....X...................................X...................] duration:
1.652183924s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:09:46 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.652254252s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..X.................................X.......................] duration:
3.011968348s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:09:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.011992668s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X.X..................................XX...................] duration:
2.598126299s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:09:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.598180314s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...X.................................X......................] duration:
2.960948686s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:09:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.961176595s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X.....XX...............................X....................] duration:
2.285634921s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:08:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.285739057s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..XX..................................XX....................] duration:
2.703434501s trace:
https://jaeger.dev.storj.io/trace/1cc17f34cdfb5991
FAIL (2025-10-31 22:08:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.703458792s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....X.................................X.....................] duration:
2.819770595s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:08:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.819805372s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [XX.....XX...............................X...................] duration:
2.611790559s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:08:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.611812736s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....X...................................X..................] duration:
1.733423223s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:08:36 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.733450409s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...XX..................................XX...................] duration:
3.867305129s trace:
https://jaeger.dev.storj.io/trace/1cc17f34cdfb5991
FAIL (2025-10-31 22:08:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.867343576s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X.X..................................XX..................] duration:
2.362462857s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:08:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.362512918s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......X...................................X.................] duration:
1.763991854s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:08:12 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.764103772s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....XX..................................XX..................] duration:
3.157570689s trace:
https://jaeger.dev.storj.io/trace/1cc17f34cdfb5991
FAIL (2025-10-31 22:08:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.157597174s
edge
gateway
us-select-1
|