Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................................X........X..........X....] duration:
3.008058267s
FAIL (2025-10-06 23:10:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 6 23:05:54.374177603 (321h35m42.177642185s) [upload=283012]: Starting upload
Oct 6 23:05:54.374258608 (321h35m42.177723263s) [upload=283012]: Done starting upload
Oct 6 23:05:54.374363362 (321h35m42.177827943s) [upload=283012]: Uploading segments...
Oct 6 23:05:54.374565084 (321h35m42.17802967s) [upload=283012]: Got next segment. Inline: false
Oct 6 23:05:54.374619416 (321h35m42.178083998s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:05:54.374625198 (321h35m42.17808979s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:05:54.494188018 (321h35m42.297652597s) [seg_pos={0 0}, upload=283012]: Begin upload segment...
Oct 6 23:05:54.494255877 (321h35m42.297720454s) [seg_pos={0 0}, upload=283012]: Done begin upload segment.
Oct 6 23:05:54.494282957 (321h35m42.297747547s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.494478695 (321h35m42.297943272s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.495695762 (321h35m42.299160339s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496054163 (321h35m42.29951874s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496262227 (321h35m42.299726804s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496659099 (321h35m42.300123684s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496874139 (321h35m42.300338718s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497269607 (321h35m42.300734191s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497383761 (321h35m42.30084834s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497743006 (321h35m42.301207591s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.503300578 (321h35m42.306765164s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.505042243 (321h35m42.308506821s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.506394686 (321h35m42.309859265s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.507734094 (321h35m42.311198672s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.509246867 (321h35m42.312711444s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.510277459 (321h35m42.313742037s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:55.132006582 (321h35m42.935471168s) [upload=283012]: Next returned nil segment
Oct 6 23:05:55.132049300 (321h35m42.935513892s) [upload=283012]: Waiting for error group managing segments...
Oct 6 23:05:55.276997326 (321h35m43.080461904s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277505636 (321h35m43.080970214s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277891232 (321h35m43.081355816s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.278214929 (321h35m43.08167952s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.282176566 (321h35m43.085641143s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283823631 (321h35m43.087288212s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283856266 (321h35m43.087320857s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284344186 (321h35m43.087808767s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284597364 (321h35m43.088061956s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.289948030 (321h35m43.09341261s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.290567653 (321h35m43.094032245s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.294140755 (321h35m43.097605346s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:06:24.065676552 (321h36m11.869141133s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:41710->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065700549 (321h36m11.869165133s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:37320->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065859567 (321h36m11.869324164s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 139.59.144.22:33926->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066062619 (321h36m11.869527291s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066099109 (321h36m11.869563688s) [seg_pos={0 0}, upload=283012]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 6 23:06:24.180602946 (321h36m11.984067532s) [upload=283012]: Done uploading segments...
Oct 6 23:06:24.180634103 (321h36m11.984098694s) [upload=283012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.008094987s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X................X......] duration:
2.613188485s
FAIL (2025-10-06 23:10:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.613222466s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [......................................X.......X.......X..X..] duration:
7.699284475s
FAIL (2025-10-06 23:09:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.699315462s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....................................X.........X.......X...X.] duration:
2.741646971s
FAIL (2025-10-06 23:09:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
em_SegmentBegin
Oct 6 23:08:40.221054472 (321h38m28.024519074s) [seg_pos={0 0}, upload=283046]: Begin upload segment...
Oct 6 23:08:40.221140743 (321h38m28.024605343s) [seg_pos={0 0}, upload=283046]: Done begin upload segment.
Oct 6 23:08:40.221179992 (321h38m28.024644596s) [noise=true , address=45.91.116.9:4279 , piece_id=NYYJ2SEHL7JCIXLQRADKI6M7VAFA4RR562R4UTOJQKVYXNPFIQQQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.221561686 (321h38m28.025026345s) [noise=true , address=45.91.116.7:4042 , piece_id=XJGKGOTSM3M2RAQJQIDUNZLITKNTXQAIO6FLEZHMT2SS6UOBCGJA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.221997629 (321h38m28.025462231s) [noise=true , address=77.90.31.14:4666 , piece_id=PH2E7YWJED4BKANBB47J6FH3G2FJ4JMUNRDNYPGCMVH3N3PVBODA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.222178974 (321h38m28.025643573s) [noise=true , address=77.90.31.14:4670 , piece_id=YL752M75GI7SX243KE6RPJOP5NNGVYGRIII6RDRCGELPFWKXTWFQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.222525559 (321h38m28.025990158s) [noise=true , address=45.91.116.12:4237 , piece_id=NQGKFSPRP2SFX4AGCQP65PTT3KE3EURM3N2WR7Y5WUBKXY73YMOQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.222640484 (321h38m28.026105083s) [noise=true , address=77.90.31.12:4597 , piece_id=FZVEBDTBYTJJNL5UPWZNU66NFCB7JCURX6B5PT63ZJTMZ5IHHGEQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.222988014 (321h38m28.026452613s) [noise=true , address=77.90.31.11:4567 , piece_id=6YVTKLJDUCAIJKNJVDIOVRY3A23HQ32ZBU5AIFXY2HQZSPQBH7AA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223000499 (321h38m28.02646509s) [noise=true , address=145.79.227.31:4826 , piece_id=OIQNDG2O7TWVOBD3S6RNZZWOUH3BSAMYMEXUQQKIJN74ERO5DYBA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223299771 (321h38m28.026764368s) [noise=true , address=145.79.227.32:4836 , piece_id=4HVPB2CGBBY53R5RCMSR4L46XDLYFQHWYNV3FLPTRKAGGAKVHJPQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223453729 (321h38m28.026918327s) [noise=true , address=145.79.227.44:4211 , piece_id=GMMDVRSGSOOP757M3YL5R3AMTSGB4VXCPJFB3R4UP6BFEBESKIDQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223602162 (321h38m28.02706674s) [noise=true , address=77.90.31.15:4729 , piece_id=MQQYYZZUMR4C5VFVYICXT4SI36TBN4FVJTCPEQZO5ANAUVM3IC5Q, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223897567 (321h38m28.027362144s) [noise=true , address=145.79.227.36:4529 , piece_id=T57W2ET5YCHYUAZDPT6J5HTQB23IWQHQ52UE7KJ5JYALKJ6QIKNQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223939713 (321h38m28.027404321s) [noise=true , address=145.79.227.19:4098 , piece_id=G46UTN55KJYTAISVC3EPQKII7OROGDXIZKISMTLA2BPRL3K3HX7A, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.224196097 (321h38m28.027660688s) [noise=true , address=145.79.227.43:4182 , piece_id=CAMXXP65XWDQMMCMMAR7RA6V4T2APXPCUC6A6YOCNJSVKVAI4M3Q, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.224688975 (321h38m28.028153577s) [noise=true , address=145.79.227.30:4765 , piece_id=JSQFUW2PB433TYWTOUH55URDJ6WRZXAYCK2GXZ2Q4DHQNKUCNQAQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.225193126 (321h38m28.028657722s) [noise=true , address=77.90.31.21:4910 , piece_id=SHFVO6QD7UXHG6AXXBIUSEIS2AQXKCVDR5VUOGJHYFY3RASYLOJA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.225676830 (321h38m28.029141431s) [noise=true , address=145.79.227.38:4996 , piece_id=BD5WCK2SAFOA2QFJAXQR47YQFZIXCBOKE4QTP5GF6K5FTUVBDJ5A, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.227266946 (321h38m28.030731538s) [noise=true , address=145.79.227.31:4830 , piece_id=GKRSJ3BK3IKSOWLT4J4OKZYDSW3Q4EC5W2ZX2EZESPJDGZUA57PQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.799357531 (321h38m28.602822123s) [upload=283046]: Next returned nil segment
Oct 6 23:08:40.799382339 (321h38m28.602846931s) [upload=283046]: Waiting for error group managing segments...
Oct 6 23:08:40.948963327 (321h38m28.752427904s) [noise=true , address=145.79.227.19:4098 , piece_id=G46UTN55KJYTAISVC3EPQKII7OROGDXIZKISMTLA2BPRL3K3HX7A, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.959420973 (321h38m28.762885561s) [noise=true , address=145.79.227.30:4765 , piece_id=JSQFUW2PB433TYWTOUH55URDJ6WRZXAYCK2GXZ2Q4DHQNKUCNQAQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.961318418 (321h38m28.764782996s) [noise=true , address=145.79.227.44:4211 , piece_id=GMMDVRSGSOOP757M3YL5R3AMTSGB4VXCPJFB3R4UP6BFEBESKIDQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.964394500 (321h38m28.767859083s) [noise=true , address=145.79.227.38:4996 , piece_id=BD5WCK2SAFOA2QFJAXQR47YQFZIXCBOKE4QTP5GF6K5FTUVBDJ5A, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.968318821 (321h38m28.7717834s) [noise=true , address=145.79.227.36:4529 , piece_id=T57W2ET5YCHYUAZDPT6J5HTQB23IWQHQ52UE7KJ5JYALKJ6QIKNQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.969548608 (321h38m28.773013193s) [noise=true , address=145.79.227.43:4182 , piece_id=CAMXXP65XWDQMMCMMAR7RA6V4T2APXPCUC6A6YOCNJSVKVAI4M3Q, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.979944778 (321h38m28.783409359s) [noise=true , address=145.79.227.32:4836 , piece_id=4HVPB2CGBBY53R5RCMSR4L46XDLYFQHWYNV3FLPTRKAGGAKVHJPQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.981446575 (321h38m28.784911157s) [noise=true , address=145.79.227.31:4826 , piece_id=OIQNDG2O7TWVOBD3S6RNZZWOUH3BSAMYMEXUQQKIJN74ERO5DYBA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:41.447405249 (321h38m29.250869827s) [noise=true , address=145.79.227.31:4830 , piece_id=GKRSJ3BK3IKSOWLT4J4OKZYDSW3Q4EC5W2ZX2EZESPJDGZUA57PQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:09:30.379124581 (321h39m18.182589158s) [noise=true , address=45.91.116.7:4042 , piece_id=XJGKGOTSM3M2RAQJQIDUNZLITKNTXQAIO6FLEZHMT2SS6UOBCGJA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:09:30.562164117 (321h39m18.36562871s) [noise=true , address=77.90.31.12:4597 , piece_id=FZVEBDTBYTJJNL5UPWZNU66NFCB7JCURX6B5PT63ZJTMZ5IHHGEQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:09:37.709003735 (321h39m25.512468338s) [noise=true , address=77.90.31.14:4666 , piece_id=PH2E7YWJED4BKANBB47J6FH3G2FJ4JMUNRDNYPGCMVH3N3PVBODA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:09:39.790162022 (321h39m27.593626605s) [noise=true , address=77.90.31.14:4670 , piece_id=YL752M75GI7SX243KE6RPJOP5NNGVYGRIII6RDRCGELPFWKXTWFQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:12LQ2JYC2GB67iqFPW69DyASSKH9yN558jGYBFd2cRN8abegiTE, address:77.90.31.14:4670): protocol: read tcp 139.59.144.22:53666->77.90.31.14:4670: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790251732 (321h39m27.593716324s) [noise=true , address=45.91.116.12:4237 , piece_id=NQGKFSPRP2SFX4AGCQP65PTT3KE3EURM3N2WR7Y5WUBKXY73YMOQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790396915 (321h39m27.593861495s) [noise=true , address=77.90.31.21:4910 , piece_id=SHFVO6QD7UXHG6AXXBIUSEIS2AQXKCVDR5VUOGJHYFY3RASYLOJA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:12onFkiP1wSRcNuzyqegL2EjfttSrQ5AMJwfp9PBn5urHQb3C2H, address:77.90.31.21:4910): protocol: read tcp 139.59.144.22:36098->77.90.31.21:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790433348 (321h39m27.593897939s) [noise=true , address=77.90.31.15:4729 , piece_id=MQQYYZZUMR4C5VFVYICXT4SI36TBN4FVJTCPEQZO5ANAUVM3IC5Q, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: rpc: dial tcp 77.90.31.15:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790575035 (321h39m27.59403963s) [noise=true , address=45.91.116.9:4279 , piece_id=NYYJ2SEHL7JCIXLQRADKI6M7VAFA4RR562R4UTOJQKVYXNPFIQQQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: read tcp 139.59.144.22:47230->45.91.116.9:4279: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790745323 (321h39m27.594209918s) [noise=true , address=77.90.31.11:4567 , piece_id=6YVTKLJDUCAIJKNJVDIOVRY3A23HQ32ZBU5AIFXY2HQZSPQBH7AA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: read tcp 139.59.144.22:48886->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790801310 (321h39m27.594265905s) [seg_pos={0 0}, upload=283046]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:09:39.903583697 (321h39m27.707048276s) [upload=283046]: Done uploading segments...
Oct 6 23:09:39.903609834 (321h39m27.707074412s) [upload=283046]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 2.741671612s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X................X.....] duration:
2.529975206s
FAIL (2025-10-06 23:09:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.52999876s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....................................X.........X.......X...X] duration:
1m0.115438566s
FAIL (2025-10-06 23:09:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
em_SegmentBegin
Oct 6 23:08:40.221054472 (321h38m28.024519074s) [seg_pos={0 0}, upload=283046]: Begin upload segment...
Oct 6 23:08:40.221140743 (321h38m28.024605343s) [seg_pos={0 0}, upload=283046]: Done begin upload segment.
Oct 6 23:08:40.221179992 (321h38m28.024644596s) [noise=true , address=45.91.116.9:4279 , piece_id=NYYJ2SEHL7JCIXLQRADKI6M7VAFA4RR562R4UTOJQKVYXNPFIQQQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.221561686 (321h38m28.025026345s) [noise=true , address=45.91.116.7:4042 , piece_id=XJGKGOTSM3M2RAQJQIDUNZLITKNTXQAIO6FLEZHMT2SS6UOBCGJA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.221997629 (321h38m28.025462231s) [noise=true , address=77.90.31.14:4666 , piece_id=PH2E7YWJED4BKANBB47J6FH3G2FJ4JMUNRDNYPGCMVH3N3PVBODA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.222178974 (321h38m28.025643573s) [noise=true , address=77.90.31.14:4670 , piece_id=YL752M75GI7SX243KE6RPJOP5NNGVYGRIII6RDRCGELPFWKXTWFQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.222525559 (321h38m28.025990158s) [noise=true , address=45.91.116.12:4237 , piece_id=NQGKFSPRP2SFX4AGCQP65PTT3KE3EURM3N2WR7Y5WUBKXY73YMOQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.222640484 (321h38m28.026105083s) [noise=true , address=77.90.31.12:4597 , piece_id=FZVEBDTBYTJJNL5UPWZNU66NFCB7JCURX6B5PT63ZJTMZ5IHHGEQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.222988014 (321h38m28.026452613s) [noise=true , address=77.90.31.11:4567 , piece_id=6YVTKLJDUCAIJKNJVDIOVRY3A23HQ32ZBU5AIFXY2HQZSPQBH7AA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223000499 (321h38m28.02646509s) [noise=true , address=145.79.227.31:4826 , piece_id=OIQNDG2O7TWVOBD3S6RNZZWOUH3BSAMYMEXUQQKIJN74ERO5DYBA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223299771 (321h38m28.026764368s) [noise=true , address=145.79.227.32:4836 , piece_id=4HVPB2CGBBY53R5RCMSR4L46XDLYFQHWYNV3FLPTRKAGGAKVHJPQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223453729 (321h38m28.026918327s) [noise=true , address=145.79.227.44:4211 , piece_id=GMMDVRSGSOOP757M3YL5R3AMTSGB4VXCPJFB3R4UP6BFEBESKIDQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223602162 (321h38m28.02706674s) [noise=true , address=77.90.31.15:4729 , piece_id=MQQYYZZUMR4C5VFVYICXT4SI36TBN4FVJTCPEQZO5ANAUVM3IC5Q, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223897567 (321h38m28.027362144s) [noise=true , address=145.79.227.36:4529 , piece_id=T57W2ET5YCHYUAZDPT6J5HTQB23IWQHQ52UE7KJ5JYALKJ6QIKNQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223939713 (321h38m28.027404321s) [noise=true , address=145.79.227.19:4098 , piece_id=G46UTN55KJYTAISVC3EPQKII7OROGDXIZKISMTLA2BPRL3K3HX7A, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.224196097 (321h38m28.027660688s) [noise=true , address=145.79.227.43:4182 , piece_id=CAMXXP65XWDQMMCMMAR7RA6V4T2APXPCUC6A6YOCNJSVKVAI4M3Q, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.224688975 (321h38m28.028153577s) [noise=true , address=145.79.227.30:4765 , piece_id=JSQFUW2PB433TYWTOUH55URDJ6WRZXAYCK2GXZ2Q4DHQNKUCNQAQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.225193126 (321h38m28.028657722s) [noise=true , address=77.90.31.21:4910 , piece_id=SHFVO6QD7UXHG6AXXBIUSEIS2AQXKCVDR5VUOGJHYFY3RASYLOJA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.225676830 (321h38m28.029141431s) [noise=true , address=145.79.227.38:4996 , piece_id=BD5WCK2SAFOA2QFJAXQR47YQFZIXCBOKE4QTP5GF6K5FTUVBDJ5A, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.227266946 (321h38m28.030731538s) [noise=true , address=145.79.227.31:4830 , piece_id=GKRSJ3BK3IKSOWLT4J4OKZYDSW3Q4EC5W2ZX2EZESPJDGZUA57PQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.799357531 (321h38m28.602822123s) [upload=283046]: Next returned nil segment
Oct 6 23:08:40.799382339 (321h38m28.602846931s) [upload=283046]: Waiting for error group managing segments...
Oct 6 23:08:40.948963327 (321h38m28.752427904s) [noise=true , address=145.79.227.19:4098 , piece_id=G46UTN55KJYTAISVC3EPQKII7OROGDXIZKISMTLA2BPRL3K3HX7A, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.959420973 (321h38m28.762885561s) [noise=true , address=145.79.227.30:4765 , piece_id=JSQFUW2PB433TYWTOUH55URDJ6WRZXAYCK2GXZ2Q4DHQNKUCNQAQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.961318418 (321h38m28.764782996s) [noise=true , address=145.79.227.44:4211 , piece_id=GMMDVRSGSOOP757M3YL5R3AMTSGB4VXCPJFB3R4UP6BFEBESKIDQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.964394500 (321h38m28.767859083s) [noise=true , address=145.79.227.38:4996 , piece_id=BD5WCK2SAFOA2QFJAXQR47YQFZIXCBOKE4QTP5GF6K5FTUVBDJ5A, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.968318821 (321h38m28.7717834s) [noise=true , address=145.79.227.36:4529 , piece_id=T57W2ET5YCHYUAZDPT6J5HTQB23IWQHQ52UE7KJ5JYALKJ6QIKNQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.969548608 (321h38m28.773013193s) [noise=true , address=145.79.227.43:4182 , piece_id=CAMXXP65XWDQMMCMMAR7RA6V4T2APXPCUC6A6YOCNJSVKVAI4M3Q, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.979944778 (321h38m28.783409359s) [noise=true , address=145.79.227.32:4836 , piece_id=4HVPB2CGBBY53R5RCMSR4L46XDLYFQHWYNV3FLPTRKAGGAKVHJPQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.981446575 (321h38m28.784911157s) [noise=true , address=145.79.227.31:4826 , piece_id=OIQNDG2O7TWVOBD3S6RNZZWOUH3BSAMYMEXUQQKIJN74ERO5DYBA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:41.447405249 (321h38m29.250869827s) [noise=true , address=145.79.227.31:4830 , piece_id=GKRSJ3BK3IKSOWLT4J4OKZYDSW3Q4EC5W2ZX2EZESPJDGZUA57PQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:09:30.379124581 (321h39m18.182589158s) [noise=true , address=45.91.116.7:4042 , piece_id=XJGKGOTSM3M2RAQJQIDUNZLITKNTXQAIO6FLEZHMT2SS6UOBCGJA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:09:30.562164117 (321h39m18.36562871s) [noise=true , address=77.90.31.12:4597 , piece_id=FZVEBDTBYTJJNL5UPWZNU66NFCB7JCURX6B5PT63ZJTMZ5IHHGEQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:09:37.709003735 (321h39m25.512468338s) [noise=true , address=77.90.31.14:4666 , piece_id=PH2E7YWJED4BKANBB47J6FH3G2FJ4JMUNRDNYPGCMVH3N3PVBODA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:09:39.790162022 (321h39m27.593626605s) [noise=true , address=77.90.31.14:4670 , piece_id=YL752M75GI7SX243KE6RPJOP5NNGVYGRIII6RDRCGELPFWKXTWFQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:12LQ2JYC2GB67iqFPW69DyASSKH9yN558jGYBFd2cRN8abegiTE, address:77.90.31.14:4670): protocol: read tcp 139.59.144.22:53666->77.90.31.14:4670: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790251732 (321h39m27.593716324s) [noise=true , address=45.91.116.12:4237 , piece_id=NQGKFSPRP2SFX4AGCQP65PTT3KE3EURM3N2WR7Y5WUBKXY73YMOQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790396915 (321h39m27.593861495s) [noise=true , address=77.90.31.21:4910 , piece_id=SHFVO6QD7UXHG6AXXBIUSEIS2AQXKCVDR5VUOGJHYFY3RASYLOJA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:12onFkiP1wSRcNuzyqegL2EjfttSrQ5AMJwfp9PBn5urHQb3C2H, address:77.90.31.21:4910): protocol: read tcp 139.59.144.22:36098->77.90.31.21:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790433348 (321h39m27.593897939s) [noise=true , address=77.90.31.15:4729 , piece_id=MQQYYZZUMR4C5VFVYICXT4SI36TBN4FVJTCPEQZO5ANAUVM3IC5Q, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: rpc: dial tcp 77.90.31.15:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790575035 (321h39m27.59403963s) [noise=true , address=45.91.116.9:4279 , piece_id=NYYJ2SEHL7JCIXLQRADKI6M7VAFA4RR562R4UTOJQKVYXNPFIQQQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: read tcp 139.59.144.22:47230->45.91.116.9:4279: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790745323 (321h39m27.594209918s) [noise=true , address=77.90.31.11:4567 , piece_id=6YVTKLJDUCAIJKNJVDIOVRY3A23HQ32ZBU5AIFXY2HQZSPQBH7AA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: read tcp 139.59.144.22:48886->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790801310 (321h39m27.594265905s) [seg_pos={0 0}, upload=283046]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:09:39.903583697 (321h39m27.707048276s) [upload=283046]: Done uploading segments...
Oct 6 23:09:39.903609834 (321h39m27.707074412s) [upload=283046]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.115461998s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.......................................X.......X.......X..X.] duration:
4.69871802s
FAIL (2025-10-06 23:09:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.698736832s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [........................................X.......X.......X..X] duration:
1m0.001519544s
FAIL (2025-10-06 23:09:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.00154344s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................................X........X..........X...] duration:
12.930176788s
FAIL (2025-10-06 23:09:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 6 23:05:54.374177603 (321h35m42.177642185s) [upload=283012]: Starting upload
Oct 6 23:05:54.374258608 (321h35m42.177723263s) [upload=283012]: Done starting upload
Oct 6 23:05:54.374363362 (321h35m42.177827943s) [upload=283012]: Uploading segments...
Oct 6 23:05:54.374565084 (321h35m42.17802967s) [upload=283012]: Got next segment. Inline: false
Oct 6 23:05:54.374619416 (321h35m42.178083998s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:05:54.374625198 (321h35m42.17808979s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:05:54.494188018 (321h35m42.297652597s) [seg_pos={0 0}, upload=283012]: Begin upload segment...
Oct 6 23:05:54.494255877 (321h35m42.297720454s) [seg_pos={0 0}, upload=283012]: Done begin upload segment.
Oct 6 23:05:54.494282957 (321h35m42.297747547s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.494478695 (321h35m42.297943272s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.495695762 (321h35m42.299160339s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496054163 (321h35m42.29951874s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496262227 (321h35m42.299726804s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496659099 (321h35m42.300123684s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496874139 (321h35m42.300338718s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497269607 (321h35m42.300734191s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497383761 (321h35m42.30084834s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497743006 (321h35m42.301207591s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.503300578 (321h35m42.306765164s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.505042243 (321h35m42.308506821s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.506394686 (321h35m42.309859265s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.507734094 (321h35m42.311198672s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.509246867 (321h35m42.312711444s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.510277459 (321h35m42.313742037s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:55.132006582 (321h35m42.935471168s) [upload=283012]: Next returned nil segment
Oct 6 23:05:55.132049300 (321h35m42.935513892s) [upload=283012]: Waiting for error group managing segments...
Oct 6 23:05:55.276997326 (321h35m43.080461904s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277505636 (321h35m43.080970214s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277891232 (321h35m43.081355816s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.278214929 (321h35m43.08167952s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.282176566 (321h35m43.085641143s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283823631 (321h35m43.087288212s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283856266 (321h35m43.087320857s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284344186 (321h35m43.087808767s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284597364 (321h35m43.088061956s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.289948030 (321h35m43.09341261s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.290567653 (321h35m43.094032245s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.294140755 (321h35m43.097605346s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:06:24.065676552 (321h36m11.869141133s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:41710->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065700549 (321h36m11.869165133s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:37320->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065859567 (321h36m11.869324164s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 139.59.144.22:33926->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066062619 (321h36m11.869527291s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066099109 (321h36m11.869563688s) [seg_pos={0 0}, upload=283012]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 6 23:06:24.180602946 (321h36m11.984067532s) [upload=283012]: Done uploading segments...
Oct 6 23:06:24.180634103 (321h36m11.984098694s) [upload=283012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 12.9302176s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................X................X....] duration:
14.789584792s
FAIL (2025-10-06 23:09:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.789613509s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X................X...] duration:
2.750005898s
FAIL (2025-10-06 23:08:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.750031683s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................................X........X..........X..] duration:
2.536227724s
FAIL (2025-10-06 23:08:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 6 23:05:54.374177603 (321h35m42.177642185s) [upload=283012]: Starting upload
Oct 6 23:05:54.374258608 (321h35m42.177723263s) [upload=283012]: Done starting upload
Oct 6 23:05:54.374363362 (321h35m42.177827943s) [upload=283012]: Uploading segments...
Oct 6 23:05:54.374565084 (321h35m42.17802967s) [upload=283012]: Got next segment. Inline: false
Oct 6 23:05:54.374619416 (321h35m42.178083998s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:05:54.374625198 (321h35m42.17808979s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:05:54.494188018 (321h35m42.297652597s) [seg_pos={0 0}, upload=283012]: Begin upload segment...
Oct 6 23:05:54.494255877 (321h35m42.297720454s) [seg_pos={0 0}, upload=283012]: Done begin upload segment.
Oct 6 23:05:54.494282957 (321h35m42.297747547s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.494478695 (321h35m42.297943272s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.495695762 (321h35m42.299160339s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496054163 (321h35m42.29951874s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496262227 (321h35m42.299726804s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496659099 (321h35m42.300123684s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496874139 (321h35m42.300338718s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497269607 (321h35m42.300734191s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497383761 (321h35m42.30084834s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497743006 (321h35m42.301207591s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.503300578 (321h35m42.306765164s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.505042243 (321h35m42.308506821s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.506394686 (321h35m42.309859265s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.507734094 (321h35m42.311198672s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.509246867 (321h35m42.312711444s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.510277459 (321h35m42.313742037s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:55.132006582 (321h35m42.935471168s) [upload=283012]: Next returned nil segment
Oct 6 23:05:55.132049300 (321h35m42.935513892s) [upload=283012]: Waiting for error group managing segments...
Oct 6 23:05:55.276997326 (321h35m43.080461904s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277505636 (321h35m43.080970214s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277891232 (321h35m43.081355816s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.278214929 (321h35m43.08167952s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.282176566 (321h35m43.085641143s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283823631 (321h35m43.087288212s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283856266 (321h35m43.087320857s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284344186 (321h35m43.087808767s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284597364 (321h35m43.088061956s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.289948030 (321h35m43.09341261s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.290567653 (321h35m43.094032245s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.294140755 (321h35m43.097605346s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:06:24.065676552 (321h36m11.869141133s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:41710->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065700549 (321h36m11.869165133s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:37320->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065859567 (321h36m11.869324164s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 139.59.144.22:33926->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066062619 (321h36m11.869527291s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066099109 (321h36m11.869563688s) [seg_pos={0 0}, upload=283012]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 6 23:06:24.180602946 (321h36m11.984067532s) [upload=283012]: Done uploading segments...
Oct 6 23:06:24.180634103 (321h36m11.984098694s) [upload=283012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.536255251s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................................X.........X.......X...] duration:
2.506138425s
FAIL (2025-10-06 23:07:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
3:05:23.914242231 (321h35m11.717706834s) [noise=true , address=145.79.227.36:4520 , piece_id=BIMEP34VU5T6W4UDC5TZJTAX2LUQ3HYP7HLZJ6EZM6TGG4YIIOBA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.914722572 (321h35m11.718187164s) [noise=true , address=77.90.31.12:4601 , piece_id=22HE54WKC35EDWGWCEIZAIDIUOSQLGGZ7S46CJHK4H5AD3PCLNJA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.915091478 (321h35m11.718556077s) [noise=true , address=45.91.116.11:4134 , piece_id=RVXTDI24F66W3663ZRXQCL3GUYPOPK32BJRR73LYYQ6OKRBMDDFA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.915538626 (321h35m11.719003227s) [noise=true , address=145.79.227.43:4188 , piece_id=33ZTIA4YSMYWOSD2EPNVRCNWBPSNIT3SQVYTDYOUVFOGGGAYE7SA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.915991992 (321h35m11.719456584s) [noise=true , address=45.91.116.8:4066 , piece_id=ISSCXGWXKPIEQEYH7UKRJK2XWGLF2AOI5CKDSRUVSKBSNVHG5FXQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.916387238 (321h35m11.719851833s) [noise=true , address=77.90.31.14:4665 , piece_id=C3LAECYC4N7PPNB6WGEYOIOZYKGRNYNL72WK3IV7TIKSHWKT4FLQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.916983443 (321h35m11.720448046s) [noise=true , address=45.91.116.14:4222 , piece_id=5ARJMMV6U453O2GQTOOX5LOK5S722VUUPYWAAFRL4FFDU3QWSGGQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.917438043 (321h35m11.720902639s) [noise=true , address=77.90.31.15:4705 , piece_id=L4XUVY2CZ2YISHE66UREINI4SHUDLH4JZPQOJ74MEGG2QHKQULIQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.917998432 (321h35m11.721463026s) [noise=true , address=145.79.227.25:4638 , piece_id=XHKMZAWGGQ6JPZZZZZPYW2CITEGJ3DROTQWOA66SSB4F62KQ32DA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.920509225 (321h35m11.723973827s) [noise=true , address=145.79.227.32:4856 , piece_id=7WPWIXWISSAQL6FPVDSKFNRVKRIPITORRQCPHY3X4WZUBJ7QMXGA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.922691626 (321h35m11.726156227s) [noise=true , address=77.90.31.19:4845 , piece_id=WVFXMJI5WI6CJ3S7OKLOGSBZTZM3Q5G5RZASJJMKWQ5XHIWLQF6Q, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.924608002 (321h35m11.728072605s) [noise=true , address=145.79.227.22:4524 , piece_id=SKNROQ7RVDK4ZEEJUFEBJ5QO4RANWOLILVAO6QCRTOROCQMP5NYQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.926488422 (321h35m11.729953024s) [noise=true , address=145.79.227.38:4984 , piece_id=RKQU4B2DQJIX6QWZOIZORBHRR3EZVADXILOCXQFE6R7DCZITDFHA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.928376157 (321h35m11.731840759s) [noise=true , address=45.91.116.12:4256 , piece_id=3FRX4QKVPULHFH6SJF7U7QDRYL332TN2NC2AOZVV65D73VA64RMQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.929215281 (321h35m11.732679858s) [noise=true , address=77.90.31.18:4835 , piece_id=3BMMI7IASZIKPGENNJPEI5RVMSU3EDPSTWEIHAP2FU6CSS7X5UZA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:24.490374704 (321h35m12.293839285s) [upload=283004]: Next returned nil segment
Oct 6 23:05:24.490394003 (321h35m12.293858599s) [upload=283004]: Waiting for error group managing segments...
Oct 6 23:05:24.642741367 (321h35m12.446205948s) [noise=true , address=145.79.227.27:4694 , piece_id=KU5HVKNGA7PWSTPJ5W5QJMCQ4VVZKEZ57E3YOHTYHHU4VPED2QRA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.644168109 (321h35m12.447632694s) [noise=true , address=145.79.227.43:4188 , piece_id=33ZTIA4YSMYWOSD2EPNVRCNWBPSNIT3SQVYTDYOUVFOGGGAYE7SA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.644940410 (321h35m12.448405006s) [noise=true , address=145.79.227.38:4984 , piece_id=RKQU4B2DQJIX6QWZOIZORBHRR3EZVADXILOCXQFE6R7DCZITDFHA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.647967218 (321h35m12.451431799s) [noise=true , address=145.79.227.34:4917 , piece_id=WJXLYXS2CAUP5KAU6GNTLDLHCU3HLCHCLGEGE7MG3DBR4CUPTPQQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.649235053 (321h35m12.452699647s) [noise=true , address=145.79.227.22:4524 , piece_id=SKNROQ7RVDK4ZEEJUFEBJ5QO4RANWOLILVAO6QCRTOROCQMP5NYQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.650212603 (321h35m12.453677188s) [noise=true , address=145.79.227.32:4856 , piece_id=7WPWIXWISSAQL6FPVDSKFNRVKRIPITORRQCPHY3X4WZUBJ7QMXGA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.656056739 (321h35m12.459521321s) [noise=true , address=145.79.227.36:4520 , piece_id=BIMEP34VU5T6W4UDC5TZJTAX2LUQ3HYP7HLZJ6EZM6TGG4YIIOBA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.659349320 (321h35m12.462813905s) [noise=true , address=145.79.227.25:4638 , piece_id=XHKMZAWGGQ6JPZZZZZPYW2CITEGJ3DROTQWOA66SSB4F62KQ32DA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:25.358979608 (321h35m13.162444186s) [noise=true , address=64.49.13.137:4497 , piece_id=NLHZNNX63LZQPRJBAO3775OB52DADGJGZI5PZ5437J3EG5JZTUBA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:06:23.512731299 (321h36m11.316195896s) [noise=true , address=77.90.31.14:4665 , piece_id=C3LAECYC4N7PPNB6WGEYOIOZYKGRNYNL72WK3IV7TIKSHWKT4FLQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, address:77.90.31.14:4665): protocol: write tcp 139.59.144.22:33720->77.90.31.14:4665: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.512761670 (321h36m11.316226266s) [noise=true , address=45.91.116.12:4256 , piece_id=3FRX4QKVPULHFH6SJF7U7QDRYL332TN2NC2AOZVV65D73VA64RMQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME, address:45.91.116.12:4256): protocol: write tcp 139.59.144.22:48414->45.91.116.12:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.512878555 (321h36m11.316343147s) [noise=true , address=45.91.116.8:4066 , piece_id=ISSCXGWXKPIEQEYH7UKRJK2XWGLF2AOI5CKDSRUVSKBSNVHG5FXQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: read tcp 139.59.144.22:33668->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513041266 (321h36m11.316505858s) [noise=true , address=45.91.116.14:4222 , piece_id=5ARJMMV6U453O2GQTOOX5LOK5S722VUUPYWAAFRL4FFDU3QWSGGQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: read tcp 139.59.144.22:40306->45.91.116.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513076255 (321h36m11.316540849s) [noise=true , address=77.90.31.12:4601 , piece_id=22HE54WKC35EDWGWCEIZAIDIUOSQLGGZ7S46CJHK4H5AD3PCLNJA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2, address:77.90.31.12:4601): protocol: write tcp 139.59.144.22:50254->77.90.31.12:4601: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513236514 (321h36m11.31670112s) [noise=true , address=77.90.31.19:4845 , piece_id=WVFXMJI5WI6CJ3S7OKLOGSBZTZM3Q5G5RZASJJMKWQ5XHIWLQF6Q, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, address:77.90.31.19:4845): protocol: read tcp 139.59.144.22:46932->77.90.31.19:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513269980 (321h36m11.316734571s) [noise=true , address=45.91.116.11:4134 , piece_id=RVXTDI24F66W3663ZRXQCL3GUYPOPK32BJRR73LYYQ6OKRBMDDFA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7, address:45.91.116.11:4134): protocol: read tcp 139.59.144.22:51124->45.91.116.11:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513376540 (321h36m11.31684113s) [noise=true , address=77.90.31.15:4705 , piece_id=L4XUVY2CZ2YISHE66UREINI4SHUDLH4JZPQOJ74MEGG2QHKQULIQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw, address:77.90.31.15:4705): protocol: read tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513399793 (321h36m11.316864386s) [noise=true , address=77.90.31.18:4835 , piece_id=3BMMI7IASZIKPGENNJPEI5RVMSU3EDPSTWEIHAP2FU6CSS7X5UZA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:37372->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513438374 (321h36m11.316902956s) [seg_pos={0 0}, upload=283004]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 6 23:06:23.624972319 (321h36m11.428436896s) [upload=283004]: Done uploading segments...
Oct 6 23:06:23.624993899 (321h36m11.428458477s) [upload=283004]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 2.506203629s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........................................X.......X.......X..] duration:
3.027580428s
FAIL (2025-10-06 23:07:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.027606472s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................................X........X..........X.] duration:
2.702633793s
FAIL (2025-10-06 23:07:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 6 23:05:54.374177603 (321h35m42.177642185s) [upload=283012]: Starting upload
Oct 6 23:05:54.374258608 (321h35m42.177723263s) [upload=283012]: Done starting upload
Oct 6 23:05:54.374363362 (321h35m42.177827943s) [upload=283012]: Uploading segments...
Oct 6 23:05:54.374565084 (321h35m42.17802967s) [upload=283012]: Got next segment. Inline: false
Oct 6 23:05:54.374619416 (321h35m42.178083998s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:05:54.374625198 (321h35m42.17808979s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:05:54.494188018 (321h35m42.297652597s) [seg_pos={0 0}, upload=283012]: Begin upload segment...
Oct 6 23:05:54.494255877 (321h35m42.297720454s) [seg_pos={0 0}, upload=283012]: Done begin upload segment.
Oct 6 23:05:54.494282957 (321h35m42.297747547s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.494478695 (321h35m42.297943272s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.495695762 (321h35m42.299160339s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496054163 (321h35m42.29951874s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496262227 (321h35m42.299726804s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496659099 (321h35m42.300123684s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496874139 (321h35m42.300338718s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497269607 (321h35m42.300734191s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497383761 (321h35m42.30084834s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497743006 (321h35m42.301207591s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.503300578 (321h35m42.306765164s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.505042243 (321h35m42.308506821s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.506394686 (321h35m42.309859265s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.507734094 (321h35m42.311198672s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.509246867 (321h35m42.312711444s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.510277459 (321h35m42.313742037s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:55.132006582 (321h35m42.935471168s) [upload=283012]: Next returned nil segment
Oct 6 23:05:55.132049300 (321h35m42.935513892s) [upload=283012]: Waiting for error group managing segments...
Oct 6 23:05:55.276997326 (321h35m43.080461904s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277505636 (321h35m43.080970214s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277891232 (321h35m43.081355816s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.278214929 (321h35m43.08167952s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.282176566 (321h35m43.085641143s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283823631 (321h35m43.087288212s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283856266 (321h35m43.087320857s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284344186 (321h35m43.087808767s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284597364 (321h35m43.088061956s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.289948030 (321h35m43.09341261s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.290567653 (321h35m43.094032245s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.294140755 (321h35m43.097605346s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:06:24.065676552 (321h36m11.869141133s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:41710->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065700549 (321h36m11.869165133s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:37320->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065859567 (321h36m11.869324164s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 139.59.144.22:33926->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066062619 (321h36m11.869527291s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066099109 (321h36m11.869563688s) [seg_pos={0 0}, upload=283012]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 6 23:06:24.180602946 (321h36m11.984067532s) [upload=283012]: Done uploading segments...
Oct 6 23:06:24.180634103 (321h36m11.984098694s) [upload=283012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.702667564s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X................X..] duration:
2.579983639s
FAIL (2025-10-06 23:07:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.580012443s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................................X.........X.......X..] duration:
2.618848249s
FAIL (2025-10-06 23:06:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
3:05:23.914242231 (321h35m11.717706834s) [noise=true , address=145.79.227.36:4520 , piece_id=BIMEP34VU5T6W4UDC5TZJTAX2LUQ3HYP7HLZJ6EZM6TGG4YIIOBA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.914722572 (321h35m11.718187164s) [noise=true , address=77.90.31.12:4601 , piece_id=22HE54WKC35EDWGWCEIZAIDIUOSQLGGZ7S46CJHK4H5AD3PCLNJA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.915091478 (321h35m11.718556077s) [noise=true , address=45.91.116.11:4134 , piece_id=RVXTDI24F66W3663ZRXQCL3GUYPOPK32BJRR73LYYQ6OKRBMDDFA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.915538626 (321h35m11.719003227s) [noise=true , address=145.79.227.43:4188 , piece_id=33ZTIA4YSMYWOSD2EPNVRCNWBPSNIT3SQVYTDYOUVFOGGGAYE7SA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.915991992 (321h35m11.719456584s) [noise=true , address=45.91.116.8:4066 , piece_id=ISSCXGWXKPIEQEYH7UKRJK2XWGLF2AOI5CKDSRUVSKBSNVHG5FXQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.916387238 (321h35m11.719851833s) [noise=true , address=77.90.31.14:4665 , piece_id=C3LAECYC4N7PPNB6WGEYOIOZYKGRNYNL72WK3IV7TIKSHWKT4FLQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.916983443 (321h35m11.720448046s) [noise=true , address=45.91.116.14:4222 , piece_id=5ARJMMV6U453O2GQTOOX5LOK5S722VUUPYWAAFRL4FFDU3QWSGGQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.917438043 (321h35m11.720902639s) [noise=true , address=77.90.31.15:4705 , piece_id=L4XUVY2CZ2YISHE66UREINI4SHUDLH4JZPQOJ74MEGG2QHKQULIQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.917998432 (321h35m11.721463026s) [noise=true , address=145.79.227.25:4638 , piece_id=XHKMZAWGGQ6JPZZZZZPYW2CITEGJ3DROTQWOA66SSB4F62KQ32DA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.920509225 (321h35m11.723973827s) [noise=true , address=145.79.227.32:4856 , piece_id=7WPWIXWISSAQL6FPVDSKFNRVKRIPITORRQCPHY3X4WZUBJ7QMXGA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.922691626 (321h35m11.726156227s) [noise=true , address=77.90.31.19:4845 , piece_id=WVFXMJI5WI6CJ3S7OKLOGSBZTZM3Q5G5RZASJJMKWQ5XHIWLQF6Q, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.924608002 (321h35m11.728072605s) [noise=true , address=145.79.227.22:4524 , piece_id=SKNROQ7RVDK4ZEEJUFEBJ5QO4RANWOLILVAO6QCRTOROCQMP5NYQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.926488422 (321h35m11.729953024s) [noise=true , address=145.79.227.38:4984 , piece_id=RKQU4B2DQJIX6QWZOIZORBHRR3EZVADXILOCXQFE6R7DCZITDFHA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.928376157 (321h35m11.731840759s) [noise=true , address=45.91.116.12:4256 , piece_id=3FRX4QKVPULHFH6SJF7U7QDRYL332TN2NC2AOZVV65D73VA64RMQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.929215281 (321h35m11.732679858s) [noise=true , address=77.90.31.18:4835 , piece_id=3BMMI7IASZIKPGENNJPEI5RVMSU3EDPSTWEIHAP2FU6CSS7X5UZA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:24.490374704 (321h35m12.293839285s) [upload=283004]: Next returned nil segment
Oct 6 23:05:24.490394003 (321h35m12.293858599s) [upload=283004]: Waiting for error group managing segments...
Oct 6 23:05:24.642741367 (321h35m12.446205948s) [noise=true , address=145.79.227.27:4694 , piece_id=KU5HVKNGA7PWSTPJ5W5QJMCQ4VVZKEZ57E3YOHTYHHU4VPED2QRA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.644168109 (321h35m12.447632694s) [noise=true , address=145.79.227.43:4188 , piece_id=33ZTIA4YSMYWOSD2EPNVRCNWBPSNIT3SQVYTDYOUVFOGGGAYE7SA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.644940410 (321h35m12.448405006s) [noise=true , address=145.79.227.38:4984 , piece_id=RKQU4B2DQJIX6QWZOIZORBHRR3EZVADXILOCXQFE6R7DCZITDFHA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.647967218 (321h35m12.451431799s) [noise=true , address=145.79.227.34:4917 , piece_id=WJXLYXS2CAUP5KAU6GNTLDLHCU3HLCHCLGEGE7MG3DBR4CUPTPQQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.649235053 (321h35m12.452699647s) [noise=true , address=145.79.227.22:4524 , piece_id=SKNROQ7RVDK4ZEEJUFEBJ5QO4RANWOLILVAO6QCRTOROCQMP5NYQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.650212603 (321h35m12.453677188s) [noise=true , address=145.79.227.32:4856 , piece_id=7WPWIXWISSAQL6FPVDSKFNRVKRIPITORRQCPHY3X4WZUBJ7QMXGA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.656056739 (321h35m12.459521321s) [noise=true , address=145.79.227.36:4520 , piece_id=BIMEP34VU5T6W4UDC5TZJTAX2LUQ3HYP7HLZJ6EZM6TGG4YIIOBA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.659349320 (321h35m12.462813905s) [noise=true , address=145.79.227.25:4638 , piece_id=XHKMZAWGGQ6JPZZZZZPYW2CITEGJ3DROTQWOA66SSB4F62KQ32DA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:25.358979608 (321h35m13.162444186s) [noise=true , address=64.49.13.137:4497 , piece_id=NLHZNNX63LZQPRJBAO3775OB52DADGJGZI5PZ5437J3EG5JZTUBA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:06:23.512731299 (321h36m11.316195896s) [noise=true , address=77.90.31.14:4665 , piece_id=C3LAECYC4N7PPNB6WGEYOIOZYKGRNYNL72WK3IV7TIKSHWKT4FLQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, address:77.90.31.14:4665): protocol: write tcp 139.59.144.22:33720->77.90.31.14:4665: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.512761670 (321h36m11.316226266s) [noise=true , address=45.91.116.12:4256 , piece_id=3FRX4QKVPULHFH6SJF7U7QDRYL332TN2NC2AOZVV65D73VA64RMQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME, address:45.91.116.12:4256): protocol: write tcp 139.59.144.22:48414->45.91.116.12:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.512878555 (321h36m11.316343147s) [noise=true , address=45.91.116.8:4066 , piece_id=ISSCXGWXKPIEQEYH7UKRJK2XWGLF2AOI5CKDSRUVSKBSNVHG5FXQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: read tcp 139.59.144.22:33668->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513041266 (321h36m11.316505858s) [noise=true , address=45.91.116.14:4222 , piece_id=5ARJMMV6U453O2GQTOOX5LOK5S722VUUPYWAAFRL4FFDU3QWSGGQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: read tcp 139.59.144.22:40306->45.91.116.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513076255 (321h36m11.316540849s) [noise=true , address=77.90.31.12:4601 , piece_id=22HE54WKC35EDWGWCEIZAIDIUOSQLGGZ7S46CJHK4H5AD3PCLNJA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2, address:77.90.31.12:4601): protocol: write tcp 139.59.144.22:50254->77.90.31.12:4601: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513236514 (321h36m11.31670112s) [noise=true , address=77.90.31.19:4845 , piece_id=WVFXMJI5WI6CJ3S7OKLOGSBZTZM3Q5G5RZASJJMKWQ5XHIWLQF6Q, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, address:77.90.31.19:4845): protocol: read tcp 139.59.144.22:46932->77.90.31.19:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513269980 (321h36m11.316734571s) [noise=true , address=45.91.116.11:4134 , piece_id=RVXTDI24F66W3663ZRXQCL3GUYPOPK32BJRR73LYYQ6OKRBMDDFA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7, address:45.91.116.11:4134): protocol: read tcp 139.59.144.22:51124->45.91.116.11:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513376540 (321h36m11.31684113s) [noise=true , address=77.90.31.15:4705 , piece_id=L4XUVY2CZ2YISHE66UREINI4SHUDLH4JZPQOJ74MEGG2QHKQULIQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw, address:77.90.31.15:4705): protocol: read tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513399793 (321h36m11.316864386s) [noise=true , address=77.90.31.18:4835 , piece_id=3BMMI7IASZIKPGENNJPEI5RVMSU3EDPSTWEIHAP2FU6CSS7X5UZA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:37372->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513438374 (321h36m11.316902956s) [seg_pos={0 0}, upload=283004]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 6 23:06:23.624972319 (321h36m11.428436896s) [upload=283004]: Done uploading segments...
Oct 6 23:06:23.624993899 (321h36m11.428458477s) [upload=283004]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 2.618882924s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........................................X.......X.......X.] duration:
11.466856451s
FAIL (2025-10-06 23:06:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.466881905s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................................X.......X.......X] duration:
1m0.000636795s
FAIL (2025-10-06 23:06:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000666351s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................................X.........X.......X.] duration:
2.710460949s
FAIL (2025-10-06 23:06:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
3:05:23.914242231 (321h35m11.717706834s) [noise=true , address=145.79.227.36:4520 , piece_id=BIMEP34VU5T6W4UDC5TZJTAX2LUQ3HYP7HLZJ6EZM6TGG4YIIOBA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.914722572 (321h35m11.718187164s) [noise=true , address=77.90.31.12:4601 , piece_id=22HE54WKC35EDWGWCEIZAIDIUOSQLGGZ7S46CJHK4H5AD3PCLNJA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.915091478 (321h35m11.718556077s) [noise=true , address=45.91.116.11:4134 , piece_id=RVXTDI24F66W3663ZRXQCL3GUYPOPK32BJRR73LYYQ6OKRBMDDFA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.915538626 (321h35m11.719003227s) [noise=true , address=145.79.227.43:4188 , piece_id=33ZTIA4YSMYWOSD2EPNVRCNWBPSNIT3SQVYTDYOUVFOGGGAYE7SA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.915991992 (321h35m11.719456584s) [noise=true , address=45.91.116.8:4066 , piece_id=ISSCXGWXKPIEQEYH7UKRJK2XWGLF2AOI5CKDSRUVSKBSNVHG5FXQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.916387238 (321h35m11.719851833s) [noise=true , address=77.90.31.14:4665 , piece_id=C3LAECYC4N7PPNB6WGEYOIOZYKGRNYNL72WK3IV7TIKSHWKT4FLQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.916983443 (321h35m11.720448046s) [noise=true , address=45.91.116.14:4222 , piece_id=5ARJMMV6U453O2GQTOOX5LOK5S722VUUPYWAAFRL4FFDU3QWSGGQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.917438043 (321h35m11.720902639s) [noise=true , address=77.90.31.15:4705 , piece_id=L4XUVY2CZ2YISHE66UREINI4SHUDLH4JZPQOJ74MEGG2QHKQULIQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.917998432 (321h35m11.721463026s) [noise=true , address=145.79.227.25:4638 , piece_id=XHKMZAWGGQ6JPZZZZZPYW2CITEGJ3DROTQWOA66SSB4F62KQ32DA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.920509225 (321h35m11.723973827s) [noise=true , address=145.79.227.32:4856 , piece_id=7WPWIXWISSAQL6FPVDSKFNRVKRIPITORRQCPHY3X4WZUBJ7QMXGA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.922691626 (321h35m11.726156227s) [noise=true , address=77.90.31.19:4845 , piece_id=WVFXMJI5WI6CJ3S7OKLOGSBZTZM3Q5G5RZASJJMKWQ5XHIWLQF6Q, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.924608002 (321h35m11.728072605s) [noise=true , address=145.79.227.22:4524 , piece_id=SKNROQ7RVDK4ZEEJUFEBJ5QO4RANWOLILVAO6QCRTOROCQMP5NYQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.926488422 (321h35m11.729953024s) [noise=true , address=145.79.227.38:4984 , piece_id=RKQU4B2DQJIX6QWZOIZORBHRR3EZVADXILOCXQFE6R7DCZITDFHA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.928376157 (321h35m11.731840759s) [noise=true , address=45.91.116.12:4256 , piece_id=3FRX4QKVPULHFH6SJF7U7QDRYL332TN2NC2AOZVV65D73VA64RMQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.929215281 (321h35m11.732679858s) [noise=true , address=77.90.31.18:4835 , piece_id=3BMMI7IASZIKPGENNJPEI5RVMSU3EDPSTWEIHAP2FU6CSS7X5UZA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:24.490374704 (321h35m12.293839285s) [upload=283004]: Next returned nil segment
Oct 6 23:05:24.490394003 (321h35m12.293858599s) [upload=283004]: Waiting for error group managing segments...
Oct 6 23:05:24.642741367 (321h35m12.446205948s) [noise=true , address=145.79.227.27:4694 , piece_id=KU5HVKNGA7PWSTPJ5W5QJMCQ4VVZKEZ57E3YOHTYHHU4VPED2QRA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.644168109 (321h35m12.447632694s) [noise=true , address=145.79.227.43:4188 , piece_id=33ZTIA4YSMYWOSD2EPNVRCNWBPSNIT3SQVYTDYOUVFOGGGAYE7SA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.644940410 (321h35m12.448405006s) [noise=true , address=145.79.227.38:4984 , piece_id=RKQU4B2DQJIX6QWZOIZORBHRR3EZVADXILOCXQFE6R7DCZITDFHA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.647967218 (321h35m12.451431799s) [noise=true , address=145.79.227.34:4917 , piece_id=WJXLYXS2CAUP5KAU6GNTLDLHCU3HLCHCLGEGE7MG3DBR4CUPTPQQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.649235053 (321h35m12.452699647s) [noise=true , address=145.79.227.22:4524 , piece_id=SKNROQ7RVDK4ZEEJUFEBJ5QO4RANWOLILVAO6QCRTOROCQMP5NYQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.650212603 (321h35m12.453677188s) [noise=true , address=145.79.227.32:4856 , piece_id=7WPWIXWISSAQL6FPVDSKFNRVKRIPITORRQCPHY3X4WZUBJ7QMXGA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.656056739 (321h35m12.459521321s) [noise=true , address=145.79.227.36:4520 , piece_id=BIMEP34VU5T6W4UDC5TZJTAX2LUQ3HYP7HLZJ6EZM6TGG4YIIOBA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.659349320 (321h35m12.462813905s) [noise=true , address=145.79.227.25:4638 , piece_id=XHKMZAWGGQ6JPZZZZZPYW2CITEGJ3DROTQWOA66SSB4F62KQ32DA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:25.358979608 (321h35m13.162444186s) [noise=true , address=64.49.13.137:4497 , piece_id=NLHZNNX63LZQPRJBAO3775OB52DADGJGZI5PZ5437J3EG5JZTUBA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:06:23.512731299 (321h36m11.316195896s) [noise=true , address=77.90.31.14:4665 , piece_id=C3LAECYC4N7PPNB6WGEYOIOZYKGRNYNL72WK3IV7TIKSHWKT4FLQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, address:77.90.31.14:4665): protocol: write tcp 139.59.144.22:33720->77.90.31.14:4665: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.512761670 (321h36m11.316226266s) [noise=true , address=45.91.116.12:4256 , piece_id=3FRX4QKVPULHFH6SJF7U7QDRYL332TN2NC2AOZVV65D73VA64RMQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME, address:45.91.116.12:4256): protocol: write tcp 139.59.144.22:48414->45.91.116.12:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.512878555 (321h36m11.316343147s) [noise=true , address=45.91.116.8:4066 , piece_id=ISSCXGWXKPIEQEYH7UKRJK2XWGLF2AOI5CKDSRUVSKBSNVHG5FXQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: read tcp 139.59.144.22:33668->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513041266 (321h36m11.316505858s) [noise=true , address=45.91.116.14:4222 , piece_id=5ARJMMV6U453O2GQTOOX5LOK5S722VUUPYWAAFRL4FFDU3QWSGGQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: read tcp 139.59.144.22:40306->45.91.116.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513076255 (321h36m11.316540849s) [noise=true , address=77.90.31.12:4601 , piece_id=22HE54WKC35EDWGWCEIZAIDIUOSQLGGZ7S46CJHK4H5AD3PCLNJA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2, address:77.90.31.12:4601): protocol: write tcp 139.59.144.22:50254->77.90.31.12:4601: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513236514 (321h36m11.31670112s) [noise=true , address=77.90.31.19:4845 , piece_id=WVFXMJI5WI6CJ3S7OKLOGSBZTZM3Q5G5RZASJJMKWQ5XHIWLQF6Q, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, address:77.90.31.19:4845): protocol: read tcp 139.59.144.22:46932->77.90.31.19:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513269980 (321h36m11.316734571s) [noise=true , address=45.91.116.11:4134 , piece_id=RVXTDI24F66W3663ZRXQCL3GUYPOPK32BJRR73LYYQ6OKRBMDDFA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7, address:45.91.116.11:4134): protocol: read tcp 139.59.144.22:51124->45.91.116.11:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513376540 (321h36m11.31684113s) [noise=true , address=77.90.31.15:4705 , piece_id=L4XUVY2CZ2YISHE66UREINI4SHUDLH4JZPQOJ74MEGG2QHKQULIQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw, address:77.90.31.15:4705): protocol: read tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513399793 (321h36m11.316864386s) [noise=true , address=77.90.31.18:4835 , piece_id=3BMMI7IASZIKPGENNJPEI5RVMSU3EDPSTWEIHAP2FU6CSS7X5UZA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:37372->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513438374 (321h36m11.316902956s) [seg_pos={0 0}, upload=283004]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 6 23:06:23.624972319 (321h36m11.428436896s) [upload=283004]: Done uploading segments...
Oct 6 23:06:23.624993899 (321h36m11.428458477s) [upload=283004]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 2.710482662s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................................X........X..........X] duration:
30.116951469s
FAIL (2025-10-06 23:06:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 6 23:05:54.374177603 (321h35m42.177642185s) [upload=283012]: Starting upload
Oct 6 23:05:54.374258608 (321h35m42.177723263s) [upload=283012]: Done starting upload
Oct 6 23:05:54.374363362 (321h35m42.177827943s) [upload=283012]: Uploading segments...
Oct 6 23:05:54.374565084 (321h35m42.17802967s) [upload=283012]: Got next segment. Inline: false
Oct 6 23:05:54.374619416 (321h35m42.178083998s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:05:54.374625198 (321h35m42.17808979s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:05:54.494188018 (321h35m42.297652597s) [seg_pos={0 0}, upload=283012]: Begin upload segment...
Oct 6 23:05:54.494255877 (321h35m42.297720454s) [seg_pos={0 0}, upload=283012]: Done begin upload segment.
Oct 6 23:05:54.494282957 (321h35m42.297747547s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.494478695 (321h35m42.297943272s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.495695762 (321h35m42.299160339s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496054163 (321h35m42.29951874s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496262227 (321h35m42.299726804s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496659099 (321h35m42.300123684s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496874139 (321h35m42.300338718s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497269607 (321h35m42.300734191s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497383761 (321h35m42.30084834s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497743006 (321h35m42.301207591s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.503300578 (321h35m42.306765164s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.505042243 (321h35m42.308506821s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.506394686 (321h35m42.309859265s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.507734094 (321h35m42.311198672s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.509246867 (321h35m42.312711444s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.510277459 (321h35m42.313742037s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:55.132006582 (321h35m42.935471168s) [upload=283012]: Next returned nil segment
Oct 6 23:05:55.132049300 (321h35m42.935513892s) [upload=283012]: Waiting for error group managing segments...
Oct 6 23:05:55.276997326 (321h35m43.080461904s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277505636 (321h35m43.080970214s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277891232 (321h35m43.081355816s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.278214929 (321h35m43.08167952s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.282176566 (321h35m43.085641143s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283823631 (321h35m43.087288212s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283856266 (321h35m43.087320857s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284344186 (321h35m43.087808767s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284597364 (321h35m43.088061956s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.289948030 (321h35m43.09341261s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.290567653 (321h35m43.094032245s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.294140755 (321h35m43.097605346s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:06:24.065676552 (321h36m11.869141133s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:41710->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065700549 (321h36m11.869165133s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:37320->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065859567 (321h36m11.869324164s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 139.59.144.22:33926->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066062619 (321h36m11.869527291s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066099109 (321h36m11.869563688s) [seg_pos={0 0}, upload=283012]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 6 23:06:24.180602946 (321h36m11.984067532s) [upload=283012]: Done uploading segments...
Oct 6 23:06:24.180634103 (321h36m11.984098694s) [upload=283012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.116976093s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................X.........X.......X] duration:
1m0.114234736s
FAIL (2025-10-06 23:06:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
3:05:23.914242231 (321h35m11.717706834s) [noise=true , address=145.79.227.36:4520 , piece_id=BIMEP34VU5T6W4UDC5TZJTAX2LUQ3HYP7HLZJ6EZM6TGG4YIIOBA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.914722572 (321h35m11.718187164s) [noise=true , address=77.90.31.12:4601 , piece_id=22HE54WKC35EDWGWCEIZAIDIUOSQLGGZ7S46CJHK4H5AD3PCLNJA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.915091478 (321h35m11.718556077s) [noise=true , address=45.91.116.11:4134 , piece_id=RVXTDI24F66W3663ZRXQCL3GUYPOPK32BJRR73LYYQ6OKRBMDDFA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.915538626 (321h35m11.719003227s) [noise=true , address=145.79.227.43:4188 , piece_id=33ZTIA4YSMYWOSD2EPNVRCNWBPSNIT3SQVYTDYOUVFOGGGAYE7SA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.915991992 (321h35m11.719456584s) [noise=true , address=45.91.116.8:4066 , piece_id=ISSCXGWXKPIEQEYH7UKRJK2XWGLF2AOI5CKDSRUVSKBSNVHG5FXQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.916387238 (321h35m11.719851833s) [noise=true , address=77.90.31.14:4665 , piece_id=C3LAECYC4N7PPNB6WGEYOIOZYKGRNYNL72WK3IV7TIKSHWKT4FLQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.916983443 (321h35m11.720448046s) [noise=true , address=45.91.116.14:4222 , piece_id=5ARJMMV6U453O2GQTOOX5LOK5S722VUUPYWAAFRL4FFDU3QWSGGQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.917438043 (321h35m11.720902639s) [noise=true , address=77.90.31.15:4705 , piece_id=L4XUVY2CZ2YISHE66UREINI4SHUDLH4JZPQOJ74MEGG2QHKQULIQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.917998432 (321h35m11.721463026s) [noise=true , address=145.79.227.25:4638 , piece_id=XHKMZAWGGQ6JPZZZZZPYW2CITEGJ3DROTQWOA66SSB4F62KQ32DA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.920509225 (321h35m11.723973827s) [noise=true , address=145.79.227.32:4856 , piece_id=7WPWIXWISSAQL6FPVDSKFNRVKRIPITORRQCPHY3X4WZUBJ7QMXGA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.922691626 (321h35m11.726156227s) [noise=true , address=77.90.31.19:4845 , piece_id=WVFXMJI5WI6CJ3S7OKLOGSBZTZM3Q5G5RZASJJMKWQ5XHIWLQF6Q, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.924608002 (321h35m11.728072605s) [noise=true , address=145.79.227.22:4524 , piece_id=SKNROQ7RVDK4ZEEJUFEBJ5QO4RANWOLILVAO6QCRTOROCQMP5NYQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.926488422 (321h35m11.729953024s) [noise=true , address=145.79.227.38:4984 , piece_id=RKQU4B2DQJIX6QWZOIZORBHRR3EZVADXILOCXQFE6R7DCZITDFHA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.928376157 (321h35m11.731840759s) [noise=true , address=45.91.116.12:4256 , piece_id=3FRX4QKVPULHFH6SJF7U7QDRYL332TN2NC2AOZVV65D73VA64RMQ, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:23.929215281 (321h35m11.732679858s) [noise=true , address=77.90.31.18:4835 , piece_id=3BMMI7IASZIKPGENNJPEI5RVMSU3EDPSTWEIHAP2FU6CSS7X5UZA, seg_pos={0 0}, upload=283004]: Uploading piece...
Oct 6 23:05:24.490374704 (321h35m12.293839285s) [upload=283004]: Next returned nil segment
Oct 6 23:05:24.490394003 (321h35m12.293858599s) [upload=283004]: Waiting for error group managing segments...
Oct 6 23:05:24.642741367 (321h35m12.446205948s) [noise=true , address=145.79.227.27:4694 , piece_id=KU5HVKNGA7PWSTPJ5W5QJMCQ4VVZKEZ57E3YOHTYHHU4VPED2QRA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.644168109 (321h35m12.447632694s) [noise=true , address=145.79.227.43:4188 , piece_id=33ZTIA4YSMYWOSD2EPNVRCNWBPSNIT3SQVYTDYOUVFOGGGAYE7SA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.644940410 (321h35m12.448405006s) [noise=true , address=145.79.227.38:4984 , piece_id=RKQU4B2DQJIX6QWZOIZORBHRR3EZVADXILOCXQFE6R7DCZITDFHA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.647967218 (321h35m12.451431799s) [noise=true , address=145.79.227.34:4917 , piece_id=WJXLYXS2CAUP5KAU6GNTLDLHCU3HLCHCLGEGE7MG3DBR4CUPTPQQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.649235053 (321h35m12.452699647s) [noise=true , address=145.79.227.22:4524 , piece_id=SKNROQ7RVDK4ZEEJUFEBJ5QO4RANWOLILVAO6QCRTOROCQMP5NYQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.650212603 (321h35m12.453677188s) [noise=true , address=145.79.227.32:4856 , piece_id=7WPWIXWISSAQL6FPVDSKFNRVKRIPITORRQCPHY3X4WZUBJ7QMXGA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.656056739 (321h35m12.459521321s) [noise=true , address=145.79.227.36:4520 , piece_id=BIMEP34VU5T6W4UDC5TZJTAX2LUQ3HYP7HLZJ6EZM6TGG4YIIOBA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:24.659349320 (321h35m12.462813905s) [noise=true , address=145.79.227.25:4638 , piece_id=XHKMZAWGGQ6JPZZZZZPYW2CITEGJ3DROTQWOA66SSB4F62KQ32DA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:05:25.358979608 (321h35m13.162444186s) [noise=true , address=64.49.13.137:4497 , piece_id=NLHZNNX63LZQPRJBAO3775OB52DADGJGZI5PZ5437J3EG5JZTUBA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: <nil>
Oct 6 23:06:23.512731299 (321h36m11.316195896s) [noise=true , address=77.90.31.14:4665 , piece_id=C3LAECYC4N7PPNB6WGEYOIOZYKGRNYNL72WK3IV7TIKSHWKT4FLQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, address:77.90.31.14:4665): protocol: write tcp 139.59.144.22:33720->77.90.31.14:4665: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.512761670 (321h36m11.316226266s) [noise=true , address=45.91.116.12:4256 , piece_id=3FRX4QKVPULHFH6SJF7U7QDRYL332TN2NC2AOZVV65D73VA64RMQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME, address:45.91.116.12:4256): protocol: write tcp 139.59.144.22:48414->45.91.116.12:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.512878555 (321h36m11.316343147s) [noise=true , address=45.91.116.8:4066 , piece_id=ISSCXGWXKPIEQEYH7UKRJK2XWGLF2AOI5CKDSRUVSKBSNVHG5FXQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: read tcp 139.59.144.22:33668->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513041266 (321h36m11.316505858s) [noise=true , address=45.91.116.14:4222 , piece_id=5ARJMMV6U453O2GQTOOX5LOK5S722VUUPYWAAFRL4FFDU3QWSGGQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: read tcp 139.59.144.22:40306->45.91.116.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513076255 (321h36m11.316540849s) [noise=true , address=77.90.31.12:4601 , piece_id=22HE54WKC35EDWGWCEIZAIDIUOSQLGGZ7S46CJHK4H5AD3PCLNJA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2, address:77.90.31.12:4601): protocol: write tcp 139.59.144.22:50254->77.90.31.12:4601: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513236514 (321h36m11.31670112s) [noise=true , address=77.90.31.19:4845 , piece_id=WVFXMJI5WI6CJ3S7OKLOGSBZTZM3Q5G5RZASJJMKWQ5XHIWLQF6Q, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, address:77.90.31.19:4845): protocol: read tcp 139.59.144.22:46932->77.90.31.19:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513269980 (321h36m11.316734571s) [noise=true , address=45.91.116.11:4134 , piece_id=RVXTDI24F66W3663ZRXQCL3GUYPOPK32BJRR73LYYQ6OKRBMDDFA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7, address:45.91.116.11:4134): protocol: read tcp 139.59.144.22:51124->45.91.116.11:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513376540 (321h36m11.31684113s) [noise=true , address=77.90.31.15:4705 , piece_id=L4XUVY2CZ2YISHE66UREINI4SHUDLH4JZPQOJ74MEGG2QHKQULIQ, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw, address:77.90.31.15:4705): protocol: read tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513399793 (321h36m11.316864386s) [noise=true , address=77.90.31.18:4835 , piece_id=3BMMI7IASZIKPGENNJPEI5RVMSU3EDPSTWEIHAP2FU6CSS7X5UZA, seg_pos={0 0}, upload=283004]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:37372->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:23.513438374 (321h36m11.316902956s) [seg_pos={0 0}, upload=283004]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 6 23:06:23.624972319 (321h36m11.428436896s) [upload=283004]: Done uploading segments...
Oct 6 23:06:23.624993899 (321h36m11.428458477s) [upload=283004]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.114259386s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X................X.] duration:
2.854029961s
FAIL (2025-10-06 23:06:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.854052228s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X................X] duration:
30.000486409s
FAIL (2025-10-06 23:06:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000505355s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............................................X.......X.......] duration:
3.034445277s
FAIL (2025-10-06 23:05:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.034460496s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X........X..........] duration:
2.624890177s
FAIL (2025-10-06 23:05:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ing segments...
Oct 6 22:59:34.773117150 (321h29m22.576581742s) [upload=282913]: Got next segment. Inline: false
Oct 6 22:59:34.773168147 (321h29m22.576632739s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:59:34.773173665 (321h29m22.576638259s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:59:34.883128853 (321h29m22.686593449s) [seg_pos={0 0}, upload=282913]: Begin upload segment...
Oct 6 22:59:34.883196370 (321h29m22.686660965s) [seg_pos={0 0}, upload=282913]: Done begin upload segment.
Oct 6 22:59:34.883221355 (321h29m22.686685953s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884198009 (321h29m22.687662601s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884476247 (321h29m22.687940838s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884797902 (321h29m22.688262496s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885129708 (321h29m22.6885943s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885392103 (321h29m22.688856694s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885650214 (321h29m22.689114805s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.894384549 (321h29m22.697849143s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897131662 (321h29m22.700596241s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897440495 (321h29m22.700905093s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.898940881 (321h29m22.702405482s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.899674583 (321h29m22.703139176s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.900689546 (321h29m22.70415414s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902009925 (321h29m22.705474516s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902390241 (321h29m22.705854818s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.903639900 (321h29m22.70710449s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:35.515159175 (321h29m23.318623786s) [upload=282913]: Next returned nil segment
Oct 6 22:59:35.515197634 (321h29m23.318662231s) [upload=282913]: Waiting for error group managing segments...
Oct 6 22:59:35.651802100 (321h29m23.455266678s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.663909364 (321h29m23.467373949s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.664794249 (321h29m23.468258831s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.669222290 (321h29m23.472686882s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675774948 (321h29m23.479239526s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675983428 (321h29m23.479448021s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.680587049 (321h29m23.484051641s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.683200435 (321h29m23.486665013s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 23:00:04.482633190 (321h29m52.286097784s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.482919071 (321h29m52.286383665s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483083379 (321h29m52.286547977s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 139.59.144.22:50470->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483213332 (321h29m52.286677931s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:55178->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483398994 (321h29m52.286863591s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:33128->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483526497 (321h29m52.286991081s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 139.59.144.22:46380->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483650237 (321h29m52.287114832s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:53980->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483806216 (321h29m52.287270812s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 139.59.144.22:41606->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483836757 (321h29m52.287301337s) [seg_pos={0 0}, upload=282913]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:00:04.589615554 (321h29m52.393080166s) [upload=282913]: Done uploading segments...
Oct 6 23:00:04.589650592 (321h29m52.393115189s) [upload=282913]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.624991418s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.845837986s
FAIL (2025-10-06 23:04:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.845868035s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X........X.........] duration:
2.706911605s
FAIL (2025-10-06 23:04:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ing segments...
Oct 6 22:59:34.773117150 (321h29m22.576581742s) [upload=282913]: Got next segment. Inline: false
Oct 6 22:59:34.773168147 (321h29m22.576632739s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:59:34.773173665 (321h29m22.576638259s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:59:34.883128853 (321h29m22.686593449s) [seg_pos={0 0}, upload=282913]: Begin upload segment...
Oct 6 22:59:34.883196370 (321h29m22.686660965s) [seg_pos={0 0}, upload=282913]: Done begin upload segment.
Oct 6 22:59:34.883221355 (321h29m22.686685953s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884198009 (321h29m22.687662601s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884476247 (321h29m22.687940838s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884797902 (321h29m22.688262496s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885129708 (321h29m22.6885943s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885392103 (321h29m22.688856694s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885650214 (321h29m22.689114805s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.894384549 (321h29m22.697849143s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897131662 (321h29m22.700596241s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897440495 (321h29m22.700905093s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.898940881 (321h29m22.702405482s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.899674583 (321h29m22.703139176s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.900689546 (321h29m22.70415414s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902009925 (321h29m22.705474516s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902390241 (321h29m22.705854818s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.903639900 (321h29m22.70710449s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:35.515159175 (321h29m23.318623786s) [upload=282913]: Next returned nil segment
Oct 6 22:59:35.515197634 (321h29m23.318662231s) [upload=282913]: Waiting for error group managing segments...
Oct 6 22:59:35.651802100 (321h29m23.455266678s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.663909364 (321h29m23.467373949s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.664794249 (321h29m23.468258831s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.669222290 (321h29m23.472686882s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675774948 (321h29m23.479239526s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675983428 (321h29m23.479448021s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.680587049 (321h29m23.484051641s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.683200435 (321h29m23.486665013s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 23:00:04.482633190 (321h29m52.286097784s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.482919071 (321h29m52.286383665s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483083379 (321h29m52.286547977s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 139.59.144.22:50470->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483213332 (321h29m52.286677931s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:55178->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483398994 (321h29m52.286863591s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:33128->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483526497 (321h29m52.286991081s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 139.59.144.22:46380->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483650237 (321h29m52.287114832s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:53980->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483806216 (321h29m52.287270812s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 139.59.144.22:41606->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483836757 (321h29m52.287301337s) [seg_pos={0 0}, upload=282913]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:00:04.589615554 (321h29m52.393080166s) [upload=282913]: Done uploading segments...
Oct 6 23:00:04.589650592 (321h29m52.393115189s) [upload=282913]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.70694644s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X.........X.......] duration:
2.484933159s
FAIL (2025-10-06 23:04:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
m12.051558028s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.248295297 (321h29m12.051759897s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249271467 (321h29m12.052736069s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249537353 (321h29m12.053001945s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250112357 (321h29m12.053576956s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250417134 (321h29m12.053881737s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250657516 (321h29m12.054122112s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250957826 (321h29m12.054422433s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251190960 (321h29m12.054655555s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251821271 (321h29m12.055285871s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.253336962 (321h29m12.056801566s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.831942906 (321h29m12.635407485s) [upload=282911]: Next returned nil segment
Oct 6 22:59:24.831960362 (321h29m12.635424956s) [upload=282911]: Waiting for error group managing segments...
Oct 6 22:59:25.014561142 (321h29m12.818025734s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.025424317 (321h29m12.828888911s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.066735306 (321h29m12.870199898s) [noise=true , address=145.79.227.30:4784 , piece_id=XLB6V3VELTPPGNT2DPO364YVME2MLBAIQIV3CEEC7GBCH3KXSV4A, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.101662779 (321h29m12.905127363s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:17.571870394 (321h30m5.375334985s) [noise=true , address=77.90.31.19:4871 , piece_id=JI63LCXG2BHAPINKCNZ3VNWR5KSXEWRX32RITFEK27HUYWWJ7MDA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:22.271026574 (321h30m10.07449117s) [noise=true , address=77.90.31.14:4666 , piece_id=QKK3OYJ5SQZBCSKRREUWT5RURTP42RKBDC46O2YDJU5EXRSFUYTQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:23.793562436 (321h30m11.59702703s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection
Oct 6 23:00:23.793610979 (321h30m11.597075579s) [noise=true , address=45.91.116.11:4120 , piece_id=UMJUJWETDHKQOCVK7PBEAFMCJKJ57VTK55MT3NYQYA5H3CWX5MXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, address:45.91.116.11:4120): protocol: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection
Oct 6 23:00:23.793846256 (321h30m11.597310852s) [noise=true , address=77.90.31.17:4790 , piece_id=JEMWDL6WHNOPB4HHATVC4DVCOC3ALP7OLNLCSNNI25KAJB2XDTBQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: write tcp 139.59.144.22:55826->77.90.31.17:4790: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.793891070 (321h30m11.59735567s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794117656 (321h30m11.597582253s) [noise=true , address=77.90.31.16:4738 , piece_id=U4ILYBFLXPMA3JTOJ3GIKG2VRHKN3WKDVFGPQ7A2CRI3H4CCW5QQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: rpc: dial tcp 77.90.31.16:4738: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794273303 (321h30m11.597737898s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794343426 (321h30m11.597808023s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:59526->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794628089 (321h30m11.598092683s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: rpc: dial tcp 77.90.31.10:4529: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794650454 (321h30m11.598115055s) [noise=true , address=45.91.116.6:4007 , piece_id=CA7D3OTYPKQBUWZDR7VDGUU5CIEBQ5YRAAT6ZZYUS733QAPKJEDQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:16iqnuPtNs9mPecc5vz71XZ3wpsjvD56UQ5sy9cs2o1uWRmNX6, address:45.91.116.6:4007): protocol: read tcp 139.59.144.22:60928->45.91.116.6:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794934492 (321h30m11.598399117s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: read tcp 139.59.144.22:55224->77.90.31.21:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794963283 (321h30m11.598427883s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA, address:45.91.116.12:4249): protocol: read tcp 139.59.144.22:52560->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.795114393 (321h30m11.598578986s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ELnn3hruPY9FSiYuYSz2bNeFRT9Lbu6nYDQEhqRsZGMZD2E9X, address:45.91.116.7:4018): protocol: read tcp 139.59.144.22:33630->45.91.116.7:4018: use of closed network connection; context deadline exceeded
Oct 6 23:00:23.795148801 (321h30m11.598613394s) [seg_pos={0 0}, upload=282911]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 23:00:23.910977478 (321h30m11.714442086s) [upload=282911]: Done uploading segments...
Oct 6 23:00:23.911014610 (321h30m11.714479191s) [upload=282911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.484962992s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X.......X......] duration:
3.054162615s
FAIL (2025-10-06 23:04:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.054183117s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X........X........] duration:
2.589924876s
FAIL (2025-10-06 23:03:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ing segments...
Oct 6 22:59:34.773117150 (321h29m22.576581742s) [upload=282913]: Got next segment. Inline: false
Oct 6 22:59:34.773168147 (321h29m22.576632739s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:59:34.773173665 (321h29m22.576638259s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:59:34.883128853 (321h29m22.686593449s) [seg_pos={0 0}, upload=282913]: Begin upload segment...
Oct 6 22:59:34.883196370 (321h29m22.686660965s) [seg_pos={0 0}, upload=282913]: Done begin upload segment.
Oct 6 22:59:34.883221355 (321h29m22.686685953s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884198009 (321h29m22.687662601s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884476247 (321h29m22.687940838s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884797902 (321h29m22.688262496s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885129708 (321h29m22.6885943s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885392103 (321h29m22.688856694s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885650214 (321h29m22.689114805s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.894384549 (321h29m22.697849143s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897131662 (321h29m22.700596241s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897440495 (321h29m22.700905093s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.898940881 (321h29m22.702405482s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.899674583 (321h29m22.703139176s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.900689546 (321h29m22.70415414s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902009925 (321h29m22.705474516s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902390241 (321h29m22.705854818s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.903639900 (321h29m22.70710449s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:35.515159175 (321h29m23.318623786s) [upload=282913]: Next returned nil segment
Oct 6 22:59:35.515197634 (321h29m23.318662231s) [upload=282913]: Waiting for error group managing segments...
Oct 6 22:59:35.651802100 (321h29m23.455266678s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.663909364 (321h29m23.467373949s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.664794249 (321h29m23.468258831s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.669222290 (321h29m23.472686882s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675774948 (321h29m23.479239526s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675983428 (321h29m23.479448021s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.680587049 (321h29m23.484051641s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.683200435 (321h29m23.486665013s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 23:00:04.482633190 (321h29m52.286097784s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.482919071 (321h29m52.286383665s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483083379 (321h29m52.286547977s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 139.59.144.22:50470->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483213332 (321h29m52.286677931s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:55178->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483398994 (321h29m52.286863591s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:33128->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483526497 (321h29m52.286991081s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 139.59.144.22:46380->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483650237 (321h29m52.287114832s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:53980->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483806216 (321h29m52.287270812s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 139.59.144.22:41606->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483836757 (321h29m52.287301337s) [seg_pos={0 0}, upload=282913]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:00:04.589615554 (321h29m52.393080166s) [upload=282913]: Done uploading segments...
Oct 6 23:00:04.589650592 (321h29m52.393115189s) [upload=282913]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.589963721s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
2.769167408s
FAIL (2025-10-06 23:03:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.769194003s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X.........X......] duration:
2.571435275s
FAIL (2025-10-06 23:03:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
m12.051558028s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.248295297 (321h29m12.051759897s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249271467 (321h29m12.052736069s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249537353 (321h29m12.053001945s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250112357 (321h29m12.053576956s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250417134 (321h29m12.053881737s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250657516 (321h29m12.054122112s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250957826 (321h29m12.054422433s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251190960 (321h29m12.054655555s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251821271 (321h29m12.055285871s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.253336962 (321h29m12.056801566s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.831942906 (321h29m12.635407485s) [upload=282911]: Next returned nil segment
Oct 6 22:59:24.831960362 (321h29m12.635424956s) [upload=282911]: Waiting for error group managing segments...
Oct 6 22:59:25.014561142 (321h29m12.818025734s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.025424317 (321h29m12.828888911s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.066735306 (321h29m12.870199898s) [noise=true , address=145.79.227.30:4784 , piece_id=XLB6V3VELTPPGNT2DPO364YVME2MLBAIQIV3CEEC7GBCH3KXSV4A, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.101662779 (321h29m12.905127363s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:17.571870394 (321h30m5.375334985s) [noise=true , address=77.90.31.19:4871 , piece_id=JI63LCXG2BHAPINKCNZ3VNWR5KSXEWRX32RITFEK27HUYWWJ7MDA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:22.271026574 (321h30m10.07449117s) [noise=true , address=77.90.31.14:4666 , piece_id=QKK3OYJ5SQZBCSKRREUWT5RURTP42RKBDC46O2YDJU5EXRSFUYTQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:23.793562436 (321h30m11.59702703s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection
Oct 6 23:00:23.793610979 (321h30m11.597075579s) [noise=true , address=45.91.116.11:4120 , piece_id=UMJUJWETDHKQOCVK7PBEAFMCJKJ57VTK55MT3NYQYA5H3CWX5MXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, address:45.91.116.11:4120): protocol: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection
Oct 6 23:00:23.793846256 (321h30m11.597310852s) [noise=true , address=77.90.31.17:4790 , piece_id=JEMWDL6WHNOPB4HHATVC4DVCOC3ALP7OLNLCSNNI25KAJB2XDTBQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: write tcp 139.59.144.22:55826->77.90.31.17:4790: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.793891070 (321h30m11.59735567s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794117656 (321h30m11.597582253s) [noise=true , address=77.90.31.16:4738 , piece_id=U4ILYBFLXPMA3JTOJ3GIKG2VRHKN3WKDVFGPQ7A2CRI3H4CCW5QQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: rpc: dial tcp 77.90.31.16:4738: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794273303 (321h30m11.597737898s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794343426 (321h30m11.597808023s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:59526->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794628089 (321h30m11.598092683s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: rpc: dial tcp 77.90.31.10:4529: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794650454 (321h30m11.598115055s) [noise=true , address=45.91.116.6:4007 , piece_id=CA7D3OTYPKQBUWZDR7VDGUU5CIEBQ5YRAAT6ZZYUS733QAPKJEDQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:16iqnuPtNs9mPecc5vz71XZ3wpsjvD56UQ5sy9cs2o1uWRmNX6, address:45.91.116.6:4007): protocol: read tcp 139.59.144.22:60928->45.91.116.6:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794934492 (321h30m11.598399117s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: read tcp 139.59.144.22:55224->77.90.31.21:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794963283 (321h30m11.598427883s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA, address:45.91.116.12:4249): protocol: read tcp 139.59.144.22:52560->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.795114393 (321h30m11.598578986s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ELnn3hruPY9FSiYuYSz2bNeFRT9Lbu6nYDQEhqRsZGMZD2E9X, address:45.91.116.7:4018): protocol: read tcp 139.59.144.22:33630->45.91.116.7:4018: use of closed network connection; context deadline exceeded
Oct 6 23:00:23.795148801 (321h30m11.598613394s) [seg_pos={0 0}, upload=282911]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 23:00:23.910977478 (321h30m11.714442086s) [upload=282911]: Done uploading segments...
Oct 6 23:00:23.911014610 (321h30m11.714479191s) [upload=282911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.571527573s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X........X.......] duration:
2.85421518s
FAIL (2025-10-06 23:03:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ing segments...
Oct 6 22:59:34.773117150 (321h29m22.576581742s) [upload=282913]: Got next segment. Inline: false
Oct 6 22:59:34.773168147 (321h29m22.576632739s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:59:34.773173665 (321h29m22.576638259s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:59:34.883128853 (321h29m22.686593449s) [seg_pos={0 0}, upload=282913]: Begin upload segment...
Oct 6 22:59:34.883196370 (321h29m22.686660965s) [seg_pos={0 0}, upload=282913]: Done begin upload segment.
Oct 6 22:59:34.883221355 (321h29m22.686685953s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884198009 (321h29m22.687662601s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884476247 (321h29m22.687940838s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884797902 (321h29m22.688262496s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885129708 (321h29m22.6885943s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885392103 (321h29m22.688856694s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885650214 (321h29m22.689114805s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.894384549 (321h29m22.697849143s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897131662 (321h29m22.700596241s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897440495 (321h29m22.700905093s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.898940881 (321h29m22.702405482s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.899674583 (321h29m22.703139176s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.900689546 (321h29m22.70415414s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902009925 (321h29m22.705474516s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902390241 (321h29m22.705854818s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.903639900 (321h29m22.70710449s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:35.515159175 (321h29m23.318623786s) [upload=282913]: Next returned nil segment
Oct 6 22:59:35.515197634 (321h29m23.318662231s) [upload=282913]: Waiting for error group managing segments...
Oct 6 22:59:35.651802100 (321h29m23.455266678s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.663909364 (321h29m23.467373949s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.664794249 (321h29m23.468258831s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.669222290 (321h29m23.472686882s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675774948 (321h29m23.479239526s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675983428 (321h29m23.479448021s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.680587049 (321h29m23.484051641s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.683200435 (321h29m23.486665013s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 23:00:04.482633190 (321h29m52.286097784s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.482919071 (321h29m52.286383665s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483083379 (321h29m52.286547977s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 139.59.144.22:50470->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483213332 (321h29m52.286677931s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:55178->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483398994 (321h29m52.286863591s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:33128->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483526497 (321h29m52.286991081s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 139.59.144.22:46380->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483650237 (321h29m52.287114832s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:53980->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483806216 (321h29m52.287270812s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 139.59.144.22:41606->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483836757 (321h29m52.287301337s) [seg_pos={0 0}, upload=282913]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:00:04.589615554 (321h29m52.393080166s) [upload=282913]: Done uploading segments...
Oct 6 23:00:04.589650592 (321h29m52.393115189s) [upload=282913]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.854249088s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
2.730744232s
FAIL (2025-10-06 23:03:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.730781935s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X.......X.....] duration:
2.803734908s
FAIL (2025-10-06 23:03:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.80377967s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X.........X.....] duration:
2.83154218s
FAIL (2025-10-06 23:03:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
m12.051558028s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.248295297 (321h29m12.051759897s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249271467 (321h29m12.052736069s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249537353 (321h29m12.053001945s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250112357 (321h29m12.053576956s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250417134 (321h29m12.053881737s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250657516 (321h29m12.054122112s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250957826 (321h29m12.054422433s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251190960 (321h29m12.054655555s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251821271 (321h29m12.055285871s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.253336962 (321h29m12.056801566s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.831942906 (321h29m12.635407485s) [upload=282911]: Next returned nil segment
Oct 6 22:59:24.831960362 (321h29m12.635424956s) [upload=282911]: Waiting for error group managing segments...
Oct 6 22:59:25.014561142 (321h29m12.818025734s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.025424317 (321h29m12.828888911s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.066735306 (321h29m12.870199898s) [noise=true , address=145.79.227.30:4784 , piece_id=XLB6V3VELTPPGNT2DPO364YVME2MLBAIQIV3CEEC7GBCH3KXSV4A, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.101662779 (321h29m12.905127363s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:17.571870394 (321h30m5.375334985s) [noise=true , address=77.90.31.19:4871 , piece_id=JI63LCXG2BHAPINKCNZ3VNWR5KSXEWRX32RITFEK27HUYWWJ7MDA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:22.271026574 (321h30m10.07449117s) [noise=true , address=77.90.31.14:4666 , piece_id=QKK3OYJ5SQZBCSKRREUWT5RURTP42RKBDC46O2YDJU5EXRSFUYTQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:23.793562436 (321h30m11.59702703s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection
Oct 6 23:00:23.793610979 (321h30m11.597075579s) [noise=true , address=45.91.116.11:4120 , piece_id=UMJUJWETDHKQOCVK7PBEAFMCJKJ57VTK55MT3NYQYA5H3CWX5MXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, address:45.91.116.11:4120): protocol: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection
Oct 6 23:00:23.793846256 (321h30m11.597310852s) [noise=true , address=77.90.31.17:4790 , piece_id=JEMWDL6WHNOPB4HHATVC4DVCOC3ALP7OLNLCSNNI25KAJB2XDTBQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: write tcp 139.59.144.22:55826->77.90.31.17:4790: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.793891070 (321h30m11.59735567s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794117656 (321h30m11.597582253s) [noise=true , address=77.90.31.16:4738 , piece_id=U4ILYBFLXPMA3JTOJ3GIKG2VRHKN3WKDVFGPQ7A2CRI3H4CCW5QQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: rpc: dial tcp 77.90.31.16:4738: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794273303 (321h30m11.597737898s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794343426 (321h30m11.597808023s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:59526->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794628089 (321h30m11.598092683s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: rpc: dial tcp 77.90.31.10:4529: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794650454 (321h30m11.598115055s) [noise=true , address=45.91.116.6:4007 , piece_id=CA7D3OTYPKQBUWZDR7VDGUU5CIEBQ5YRAAT6ZZYUS733QAPKJEDQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:16iqnuPtNs9mPecc5vz71XZ3wpsjvD56UQ5sy9cs2o1uWRmNX6, address:45.91.116.6:4007): protocol: read tcp 139.59.144.22:60928->45.91.116.6:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794934492 (321h30m11.598399117s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: read tcp 139.59.144.22:55224->77.90.31.21:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794963283 (321h30m11.598427883s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA, address:45.91.116.12:4249): protocol: read tcp 139.59.144.22:52560->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.795114393 (321h30m11.598578986s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ELnn3hruPY9FSiYuYSz2bNeFRT9Lbu6nYDQEhqRsZGMZD2E9X, address:45.91.116.7:4018): protocol: read tcp 139.59.144.22:33630->45.91.116.7:4018: use of closed network connection; context deadline exceeded
Oct 6 23:00:23.795148801 (321h30m11.598613394s) [seg_pos={0 0}, upload=282911]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 23:00:23.910977478 (321h30m11.714442086s) [upload=282911]: Done uploading segments...
Oct 6 23:00:23.911014610 (321h30m11.714479191s) [upload=282911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.831571725s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X.......X....] duration:
3.681026734s
FAIL (2025-10-06 23:03:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.681053064s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X........X......] duration:
2.731572454s
FAIL (2025-10-06 23:02:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ing segments...
Oct 6 22:59:34.773117150 (321h29m22.576581742s) [upload=282913]: Got next segment. Inline: false
Oct 6 22:59:34.773168147 (321h29m22.576632739s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:59:34.773173665 (321h29m22.576638259s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:59:34.883128853 (321h29m22.686593449s) [seg_pos={0 0}, upload=282913]: Begin upload segment...
Oct 6 22:59:34.883196370 (321h29m22.686660965s) [seg_pos={0 0}, upload=282913]: Done begin upload segment.
Oct 6 22:59:34.883221355 (321h29m22.686685953s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884198009 (321h29m22.687662601s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884476247 (321h29m22.687940838s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884797902 (321h29m22.688262496s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885129708 (321h29m22.6885943s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885392103 (321h29m22.688856694s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885650214 (321h29m22.689114805s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.894384549 (321h29m22.697849143s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897131662 (321h29m22.700596241s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897440495 (321h29m22.700905093s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.898940881 (321h29m22.702405482s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.899674583 (321h29m22.703139176s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.900689546 (321h29m22.70415414s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902009925 (321h29m22.705474516s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902390241 (321h29m22.705854818s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.903639900 (321h29m22.70710449s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:35.515159175 (321h29m23.318623786s) [upload=282913]: Next returned nil segment
Oct 6 22:59:35.515197634 (321h29m23.318662231s) [upload=282913]: Waiting for error group managing segments...
Oct 6 22:59:35.651802100 (321h29m23.455266678s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.663909364 (321h29m23.467373949s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.664794249 (321h29m23.468258831s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.669222290 (321h29m23.472686882s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675774948 (321h29m23.479239526s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675983428 (321h29m23.479448021s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.680587049 (321h29m23.484051641s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.683200435 (321h29m23.486665013s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 23:00:04.482633190 (321h29m52.286097784s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.482919071 (321h29m52.286383665s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483083379 (321h29m52.286547977s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 139.59.144.22:50470->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483213332 (321h29m52.286677931s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:55178->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483398994 (321h29m52.286863591s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:33128->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483526497 (321h29m52.286991081s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 139.59.144.22:46380->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483650237 (321h29m52.287114832s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:53980->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483806216 (321h29m52.287270812s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 139.59.144.22:41606->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483836757 (321h29m52.287301337s) [seg_pos={0 0}, upload=282913]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:00:04.589615554 (321h29m52.393080166s) [upload=282913]: Done uploading segments...
Oct 6 23:00:04.589650592 (321h29m52.393115189s) [upload=282913]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.731603831s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
2.738519009s
FAIL (2025-10-06 23:02:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.738543721s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X........X.....] duration:
2.677621404s
FAIL (2025-10-06 23:02:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ing segments...
Oct 6 22:59:34.773117150 (321h29m22.576581742s) [upload=282913]: Got next segment. Inline: false
Oct 6 22:59:34.773168147 (321h29m22.576632739s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:59:34.773173665 (321h29m22.576638259s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:59:34.883128853 (321h29m22.686593449s) [seg_pos={0 0}, upload=282913]: Begin upload segment...
Oct 6 22:59:34.883196370 (321h29m22.686660965s) [seg_pos={0 0}, upload=282913]: Done begin upload segment.
Oct 6 22:59:34.883221355 (321h29m22.686685953s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884198009 (321h29m22.687662601s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884476247 (321h29m22.687940838s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884797902 (321h29m22.688262496s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885129708 (321h29m22.6885943s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885392103 (321h29m22.688856694s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885650214 (321h29m22.689114805s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.894384549 (321h29m22.697849143s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897131662 (321h29m22.700596241s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897440495 (321h29m22.700905093s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.898940881 (321h29m22.702405482s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.899674583 (321h29m22.703139176s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.900689546 (321h29m22.70415414s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902009925 (321h29m22.705474516s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902390241 (321h29m22.705854818s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.903639900 (321h29m22.70710449s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:35.515159175 (321h29m23.318623786s) [upload=282913]: Next returned nil segment
Oct 6 22:59:35.515197634 (321h29m23.318662231s) [upload=282913]: Waiting for error group managing segments...
Oct 6 22:59:35.651802100 (321h29m23.455266678s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.663909364 (321h29m23.467373949s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.664794249 (321h29m23.468258831s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.669222290 (321h29m23.472686882s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675774948 (321h29m23.479239526s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675983428 (321h29m23.479448021s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.680587049 (321h29m23.484051641s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.683200435 (321h29m23.486665013s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 23:00:04.482633190 (321h29m52.286097784s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.482919071 (321h29m52.286383665s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483083379 (321h29m52.286547977s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 139.59.144.22:50470->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483213332 (321h29m52.286677931s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:55178->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483398994 (321h29m52.286863591s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:33128->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483526497 (321h29m52.286991081s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 139.59.144.22:46380->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483650237 (321h29m52.287114832s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:53980->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483806216 (321h29m52.287270812s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 139.59.144.22:41606->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483836757 (321h29m52.287301337s) [seg_pos={0 0}, upload=282913]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:00:04.589615554 (321h29m52.393080166s) [upload=282913]: Done uploading segments...
Oct 6 23:00:04.589650592 (321h29m52.393115189s) [upload=282913]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.677695906s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X.........X....] duration:
2.542813737s
FAIL (2025-10-06 23:02:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
m12.051558028s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.248295297 (321h29m12.051759897s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249271467 (321h29m12.052736069s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249537353 (321h29m12.053001945s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250112357 (321h29m12.053576956s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250417134 (321h29m12.053881737s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250657516 (321h29m12.054122112s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250957826 (321h29m12.054422433s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251190960 (321h29m12.054655555s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251821271 (321h29m12.055285871s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.253336962 (321h29m12.056801566s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.831942906 (321h29m12.635407485s) [upload=282911]: Next returned nil segment
Oct 6 22:59:24.831960362 (321h29m12.635424956s) [upload=282911]: Waiting for error group managing segments...
Oct 6 22:59:25.014561142 (321h29m12.818025734s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.025424317 (321h29m12.828888911s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.066735306 (321h29m12.870199898s) [noise=true , address=145.79.227.30:4784 , piece_id=XLB6V3VELTPPGNT2DPO364YVME2MLBAIQIV3CEEC7GBCH3KXSV4A, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.101662779 (321h29m12.905127363s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:17.571870394 (321h30m5.375334985s) [noise=true , address=77.90.31.19:4871 , piece_id=JI63LCXG2BHAPINKCNZ3VNWR5KSXEWRX32RITFEK27HUYWWJ7MDA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:22.271026574 (321h30m10.07449117s) [noise=true , address=77.90.31.14:4666 , piece_id=QKK3OYJ5SQZBCSKRREUWT5RURTP42RKBDC46O2YDJU5EXRSFUYTQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:23.793562436 (321h30m11.59702703s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection
Oct 6 23:00:23.793610979 (321h30m11.597075579s) [noise=true , address=45.91.116.11:4120 , piece_id=UMJUJWETDHKQOCVK7PBEAFMCJKJ57VTK55MT3NYQYA5H3CWX5MXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, address:45.91.116.11:4120): protocol: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection
Oct 6 23:00:23.793846256 (321h30m11.597310852s) [noise=true , address=77.90.31.17:4790 , piece_id=JEMWDL6WHNOPB4HHATVC4DVCOC3ALP7OLNLCSNNI25KAJB2XDTBQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: write tcp 139.59.144.22:55826->77.90.31.17:4790: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.793891070 (321h30m11.59735567s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794117656 (321h30m11.597582253s) [noise=true , address=77.90.31.16:4738 , piece_id=U4ILYBFLXPMA3JTOJ3GIKG2VRHKN3WKDVFGPQ7A2CRI3H4CCW5QQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: rpc: dial tcp 77.90.31.16:4738: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794273303 (321h30m11.597737898s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794343426 (321h30m11.597808023s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:59526->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794628089 (321h30m11.598092683s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: rpc: dial tcp 77.90.31.10:4529: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794650454 (321h30m11.598115055s) [noise=true , address=45.91.116.6:4007 , piece_id=CA7D3OTYPKQBUWZDR7VDGUU5CIEBQ5YRAAT6ZZYUS733QAPKJEDQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:16iqnuPtNs9mPecc5vz71XZ3wpsjvD56UQ5sy9cs2o1uWRmNX6, address:45.91.116.6:4007): protocol: read tcp 139.59.144.22:60928->45.91.116.6:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794934492 (321h30m11.598399117s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: read tcp 139.59.144.22:55224->77.90.31.21:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794963283 (321h30m11.598427883s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA, address:45.91.116.12:4249): protocol: read tcp 139.59.144.22:52560->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.795114393 (321h30m11.598578986s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ELnn3hruPY9FSiYuYSz2bNeFRT9Lbu6nYDQEhqRsZGMZD2E9X, address:45.91.116.7:4018): protocol: read tcp 139.59.144.22:33630->45.91.116.7:4018: use of closed network connection; context deadline exceeded
Oct 6 23:00:23.795148801 (321h30m11.598613394s) [seg_pos={0 0}, upload=282911]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 23:00:23.910977478 (321h30m11.714442086s) [upload=282911]: Done uploading segments...
Oct 6 23:00:23.911014610 (321h30m11.714479191s) [upload=282911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.542845361s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
2.678301765s
FAIL (2025-10-06 23:02:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.678331014s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X........X....] duration:
2.66938872s
FAIL (2025-10-06 23:02:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ing segments...
Oct 6 22:59:34.773117150 (321h29m22.576581742s) [upload=282913]: Got next segment. Inline: false
Oct 6 22:59:34.773168147 (321h29m22.576632739s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:59:34.773173665 (321h29m22.576638259s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:59:34.883128853 (321h29m22.686593449s) [seg_pos={0 0}, upload=282913]: Begin upload segment...
Oct 6 22:59:34.883196370 (321h29m22.686660965s) [seg_pos={0 0}, upload=282913]: Done begin upload segment.
Oct 6 22:59:34.883221355 (321h29m22.686685953s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884198009 (321h29m22.687662601s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884476247 (321h29m22.687940838s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884797902 (321h29m22.688262496s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885129708 (321h29m22.6885943s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885392103 (321h29m22.688856694s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885650214 (321h29m22.689114805s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.894384549 (321h29m22.697849143s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897131662 (321h29m22.700596241s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897440495 (321h29m22.700905093s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.898940881 (321h29m22.702405482s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.899674583 (321h29m22.703139176s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.900689546 (321h29m22.70415414s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902009925 (321h29m22.705474516s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902390241 (321h29m22.705854818s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.903639900 (321h29m22.70710449s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:35.515159175 (321h29m23.318623786s) [upload=282913]: Next returned nil segment
Oct 6 22:59:35.515197634 (321h29m23.318662231s) [upload=282913]: Waiting for error group managing segments...
Oct 6 22:59:35.651802100 (321h29m23.455266678s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.663909364 (321h29m23.467373949s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.664794249 (321h29m23.468258831s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.669222290 (321h29m23.472686882s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675774948 (321h29m23.479239526s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675983428 (321h29m23.479448021s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.680587049 (321h29m23.484051641s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.683200435 (321h29m23.486665013s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 23:00:04.482633190 (321h29m52.286097784s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.482919071 (321h29m52.286383665s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483083379 (321h29m52.286547977s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 139.59.144.22:50470->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483213332 (321h29m52.286677931s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:55178->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483398994 (321h29m52.286863591s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:33128->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483526497 (321h29m52.286991081s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 139.59.144.22:46380->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483650237 (321h29m52.287114832s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:53980->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483806216 (321h29m52.287270812s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 139.59.144.22:41606->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483836757 (321h29m52.287301337s) [seg_pos={0 0}, upload=282913]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:00:04.589615554 (321h29m52.393080166s) [upload=282913]: Done uploading segments...
Oct 6 23:00:04.589650592 (321h29m52.393115189s) [upload=282913]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.669421752s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................................X.......X...] duration:
3.004211787s
FAIL (2025-10-06 23:02:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.004237941s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X.........X...] duration:
2.573818251s
FAIL (2025-10-06 23:02:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
m12.051558028s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.248295297 (321h29m12.051759897s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249271467 (321h29m12.052736069s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249537353 (321h29m12.053001945s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250112357 (321h29m12.053576956s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250417134 (321h29m12.053881737s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250657516 (321h29m12.054122112s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250957826 (321h29m12.054422433s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251190960 (321h29m12.054655555s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251821271 (321h29m12.055285871s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.253336962 (321h29m12.056801566s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.831942906 (321h29m12.635407485s) [upload=282911]: Next returned nil segment
Oct 6 22:59:24.831960362 (321h29m12.635424956s) [upload=282911]: Waiting for error group managing segments...
Oct 6 22:59:25.014561142 (321h29m12.818025734s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.025424317 (321h29m12.828888911s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.066735306 (321h29m12.870199898s) [noise=true , address=145.79.227.30:4784 , piece_id=XLB6V3VELTPPGNT2DPO364YVME2MLBAIQIV3CEEC7GBCH3KXSV4A, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.101662779 (321h29m12.905127363s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:17.571870394 (321h30m5.375334985s) [noise=true , address=77.90.31.19:4871 , piece_id=JI63LCXG2BHAPINKCNZ3VNWR5KSXEWRX32RITFEK27HUYWWJ7MDA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:22.271026574 (321h30m10.07449117s) [noise=true , address=77.90.31.14:4666 , piece_id=QKK3OYJ5SQZBCSKRREUWT5RURTP42RKBDC46O2YDJU5EXRSFUYTQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:23.793562436 (321h30m11.59702703s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection
Oct 6 23:00:23.793610979 (321h30m11.597075579s) [noise=true , address=45.91.116.11:4120 , piece_id=UMJUJWETDHKQOCVK7PBEAFMCJKJ57VTK55MT3NYQYA5H3CWX5MXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, address:45.91.116.11:4120): protocol: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection
Oct 6 23:00:23.793846256 (321h30m11.597310852s) [noise=true , address=77.90.31.17:4790 , piece_id=JEMWDL6WHNOPB4HHATVC4DVCOC3ALP7OLNLCSNNI25KAJB2XDTBQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: write tcp 139.59.144.22:55826->77.90.31.17:4790: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.793891070 (321h30m11.59735567s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794117656 (321h30m11.597582253s) [noise=true , address=77.90.31.16:4738 , piece_id=U4ILYBFLXPMA3JTOJ3GIKG2VRHKN3WKDVFGPQ7A2CRI3H4CCW5QQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: rpc: dial tcp 77.90.31.16:4738: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794273303 (321h30m11.597737898s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794343426 (321h30m11.597808023s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:59526->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794628089 (321h30m11.598092683s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: rpc: dial tcp 77.90.31.10:4529: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794650454 (321h30m11.598115055s) [noise=true , address=45.91.116.6:4007 , piece_id=CA7D3OTYPKQBUWZDR7VDGUU5CIEBQ5YRAAT6ZZYUS733QAPKJEDQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:16iqnuPtNs9mPecc5vz71XZ3wpsjvD56UQ5sy9cs2o1uWRmNX6, address:45.91.116.6:4007): protocol: read tcp 139.59.144.22:60928->45.91.116.6:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794934492 (321h30m11.598399117s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: read tcp 139.59.144.22:55224->77.90.31.21:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794963283 (321h30m11.598427883s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA, address:45.91.116.12:4249): protocol: read tcp 139.59.144.22:52560->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.795114393 (321h30m11.598578986s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ELnn3hruPY9FSiYuYSz2bNeFRT9Lbu6nYDQEhqRsZGMZD2E9X, address:45.91.116.7:4018): protocol: read tcp 139.59.144.22:33630->45.91.116.7:4018: use of closed network connection; context deadline exceeded
Oct 6 23:00:23.795148801 (321h30m11.598613394s) [seg_pos={0 0}, upload=282911]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 23:00:23.910977478 (321h30m11.714442086s) [upload=282911]: Done uploading segments...
Oct 6 23:00:23.911014610 (321h30m11.714479191s) [upload=282911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.573869621s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X........X...] duration:
2.574387054s
FAIL (2025-10-06 23:01:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ing segments...
Oct 6 22:59:34.773117150 (321h29m22.576581742s) [upload=282913]: Got next segment. Inline: false
Oct 6 22:59:34.773168147 (321h29m22.576632739s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:59:34.773173665 (321h29m22.576638259s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:59:34.883128853 (321h29m22.686593449s) [seg_pos={0 0}, upload=282913]: Begin upload segment...
Oct 6 22:59:34.883196370 (321h29m22.686660965s) [seg_pos={0 0}, upload=282913]: Done begin upload segment.
Oct 6 22:59:34.883221355 (321h29m22.686685953s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884198009 (321h29m22.687662601s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884476247 (321h29m22.687940838s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884797902 (321h29m22.688262496s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885129708 (321h29m22.6885943s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885392103 (321h29m22.688856694s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885650214 (321h29m22.689114805s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.894384549 (321h29m22.697849143s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897131662 (321h29m22.700596241s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897440495 (321h29m22.700905093s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.898940881 (321h29m22.702405482s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.899674583 (321h29m22.703139176s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.900689546 (321h29m22.70415414s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902009925 (321h29m22.705474516s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902390241 (321h29m22.705854818s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.903639900 (321h29m22.70710449s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:35.515159175 (321h29m23.318623786s) [upload=282913]: Next returned nil segment
Oct 6 22:59:35.515197634 (321h29m23.318662231s) [upload=282913]: Waiting for error group managing segments...
Oct 6 22:59:35.651802100 (321h29m23.455266678s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.663909364 (321h29m23.467373949s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.664794249 (321h29m23.468258831s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.669222290 (321h29m23.472686882s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675774948 (321h29m23.479239526s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675983428 (321h29m23.479448021s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.680587049 (321h29m23.484051641s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.683200435 (321h29m23.486665013s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 23:00:04.482633190 (321h29m52.286097784s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.482919071 (321h29m52.286383665s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483083379 (321h29m52.286547977s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 139.59.144.22:50470->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483213332 (321h29m52.286677931s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:55178->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483398994 (321h29m52.286863591s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:33128->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483526497 (321h29m52.286991081s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 139.59.144.22:46380->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483650237 (321h29m52.287114832s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:53980->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483806216 (321h29m52.287270812s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 139.59.144.22:41606->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483836757 (321h29m52.287301337s) [seg_pos={0 0}, upload=282913]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:00:04.589615554 (321h29m52.393080166s) [upload=282913]: Done uploading segments...
Oct 6 23:00:04.589650592 (321h29m52.393115189s) [upload=282913]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.574424493s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
2.700059625s
FAIL (2025-10-06 23:01:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.700092523s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X.........X..] duration:
2.576758351s
FAIL (2025-10-06 23:01:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
m12.051558028s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.248295297 (321h29m12.051759897s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249271467 (321h29m12.052736069s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249537353 (321h29m12.053001945s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250112357 (321h29m12.053576956s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250417134 (321h29m12.053881737s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250657516 (321h29m12.054122112s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250957826 (321h29m12.054422433s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251190960 (321h29m12.054655555s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251821271 (321h29m12.055285871s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.253336962 (321h29m12.056801566s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.831942906 (321h29m12.635407485s) [upload=282911]: Next returned nil segment
Oct 6 22:59:24.831960362 (321h29m12.635424956s) [upload=282911]: Waiting for error group managing segments...
Oct 6 22:59:25.014561142 (321h29m12.818025734s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.025424317 (321h29m12.828888911s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.066735306 (321h29m12.870199898s) [noise=true , address=145.79.227.30:4784 , piece_id=XLB6V3VELTPPGNT2DPO364YVME2MLBAIQIV3CEEC7GBCH3KXSV4A, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.101662779 (321h29m12.905127363s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:17.571870394 (321h30m5.375334985s) [noise=true , address=77.90.31.19:4871 , piece_id=JI63LCXG2BHAPINKCNZ3VNWR5KSXEWRX32RITFEK27HUYWWJ7MDA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:22.271026574 (321h30m10.07449117s) [noise=true , address=77.90.31.14:4666 , piece_id=QKK3OYJ5SQZBCSKRREUWT5RURTP42RKBDC46O2YDJU5EXRSFUYTQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:23.793562436 (321h30m11.59702703s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection
Oct 6 23:00:23.793610979 (321h30m11.597075579s) [noise=true , address=45.91.116.11:4120 , piece_id=UMJUJWETDHKQOCVK7PBEAFMCJKJ57VTK55MT3NYQYA5H3CWX5MXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, address:45.91.116.11:4120): protocol: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection
Oct 6 23:00:23.793846256 (321h30m11.597310852s) [noise=true , address=77.90.31.17:4790 , piece_id=JEMWDL6WHNOPB4HHATVC4DVCOC3ALP7OLNLCSNNI25KAJB2XDTBQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: write tcp 139.59.144.22:55826->77.90.31.17:4790: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.793891070 (321h30m11.59735567s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794117656 (321h30m11.597582253s) [noise=true , address=77.90.31.16:4738 , piece_id=U4ILYBFLXPMA3JTOJ3GIKG2VRHKN3WKDVFGPQ7A2CRI3H4CCW5QQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: rpc: dial tcp 77.90.31.16:4738: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794273303 (321h30m11.597737898s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794343426 (321h30m11.597808023s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:59526->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794628089 (321h30m11.598092683s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: rpc: dial tcp 77.90.31.10:4529: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794650454 (321h30m11.598115055s) [noise=true , address=45.91.116.6:4007 , piece_id=CA7D3OTYPKQBUWZDR7VDGUU5CIEBQ5YRAAT6ZZYUS733QAPKJEDQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:16iqnuPtNs9mPecc5vz71XZ3wpsjvD56UQ5sy9cs2o1uWRmNX6, address:45.91.116.6:4007): protocol: read tcp 139.59.144.22:60928->45.91.116.6:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794934492 (321h30m11.598399117s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: read tcp 139.59.144.22:55224->77.90.31.21:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794963283 (321h30m11.598427883s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA, address:45.91.116.12:4249): protocol: read tcp 139.59.144.22:52560->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.795114393 (321h30m11.598578986s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ELnn3hruPY9FSiYuYSz2bNeFRT9Lbu6nYDQEhqRsZGMZD2E9X, address:45.91.116.7:4018): protocol: read tcp 139.59.144.22:33630->45.91.116.7:4018: use of closed network connection; context deadline exceeded
Oct 6 23:00:23.795148801 (321h30m11.598613394s) [seg_pos={0 0}, upload=282911]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 23:00:23.910977478 (321h30m11.714442086s) [upload=282911]: Done uploading segments...
Oct 6 23:00:23.911014610 (321h30m11.714479191s) [upload=282911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.576790721s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X.......X..] duration:
3.770218279s
FAIL (2025-10-06 23:01:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.770241421s
core
uplink
us-select-1
us1-placement-20
|