Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............X..............................................] duration:
1.993659383s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:29:31 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.994345119s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..........XX.......................XXXXXXX....XX............] duration:
5.470751533s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:29:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.470809275s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........XX..................................................] duration:
2.391550843s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:29:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.391574321s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........XX.................................................] duration:
3.179051074s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:29:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.17907776s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...........XX.......................XXXXXXX....XX...........] duration:
21.88942565s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:29:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 21.889451303s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
2.781206889s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:28:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.781254731s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
3.814201829s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:28:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.814233397s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........XX................................................] duration:
3.445171068s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:28:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.445209074s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............X.............................................] duration:
2.431324955s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:28:23 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.431350702s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........XX...............................................] duration:
2.605066127s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:27:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.60509342s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
3.020696015s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:27:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.020726729s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
2.235586819s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:27:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.235617902s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [............XX.......................XXXXXXX....XX..........] duration:
3.77413959s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:27:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.774214221s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.............XX.......................XXXXXXX....XX.........] duration:
4.238834461s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:27:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.238864652s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............X............................................] duration:
1.677353578s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:27:12 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.67742381s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
4.355750524s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:27:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.355784676s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..............XX.......................XXXXXXX....XX........] duration:
28.273986551s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:26:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 28.274078581s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............XX..............................................] duration:
3.507909624s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:26:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.507944601s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
3.548910621s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:26:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 3.548997486s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................X...........................................] duration:
1.785337089s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:26:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.785365162s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
3.92332316s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:26:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 3.923346948s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............XX.............................................] duration:
2.949575427s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:26:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.949622033s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................X..........................................] duration:
1.999226081s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:26:03 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.999258946s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
4.610815115s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:25:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.610847535s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...............XX.......................XXXXXXX....XX.......] duration:
18.553385683s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:25:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.553416219s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
9.941588418s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:25:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.941608095s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................X.........................................] duration:
2.118264823s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:25:41 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.118287355s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............XX............................................] duration:
4.598051145s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:25:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 4.598080264s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................X........................................] duration:
3.217746517s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:25:10 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.217770967s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
2.626461243s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:25:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.62649437s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
4.906003166s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:24:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.906074555s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............XX...........................................] duration:
3.067534193s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:24:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.067561716s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
2.457778265s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:24:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.457807706s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [................XX.......................XXXXXXX....XX......] duration:
8.181990045s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:24:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.182026773s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................XX..........................................] duration:
2.897956774s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:24:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.8979801s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................X.......................................] duration:
1.883845908s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:24:27 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.883878446s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
2.368182816s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:24:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.368210175s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................X......................................] duration:
2.80519291s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:24:03 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.805230069s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
4.635632733s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:23:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.635658062s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.................XX.......................XXXXXXX....XX.....] duration:
6.369377481s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:23:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.369479825s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................XX.........................................] duration:
2.517631293s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:23:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.517658642s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................X.....................................] duration:
1.631280434s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:23:26 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.631327253s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
2.955839093s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:23:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.955871869s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................XX........................................] duration:
2.394487323s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:23:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.39452182s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..................XX.......................XXXXXXX....XX....] duration:
9.602059371s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:23:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.6020877s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
2.566396714s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:23:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.566490125s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................XX.......................................] duration:
2.639653992s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:22:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.639682977s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
4.266198468s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:22:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.266221919s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................XX......................................] duration:
4.41154925s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:22:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 4.411571502s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......................X....................................] duration:
1.784413636s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:22:17 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.784446731s
core
uplink
us-select-1
us1-placement-20
|