Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X............................X..........................] duration: 3.243348405s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:32:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.243399565s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 2.155906305s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:32:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.155940569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X............................X.........................] duration: 4.292601767s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:31:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.292633692s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 1.606464152s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:31:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.606506999s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 2.069228057s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:30:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.069269539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X............................X........................] duration: 3.251018927s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:30:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.251050027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X............................X.......................] duration: 3.668454311s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:29:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.66848483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 2.089930397s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:29:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.089955807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 1.650475514s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:29:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.650494741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X............................X......................] duration: 3.544924528s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:29:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.544960561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X............................X.....................] duration: 3.114441947s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:28:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.114484049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X............................X....................] duration: 3.065839351s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:28:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.065884599s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 1.571064017s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:28:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.571090555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..........X............................X...................] duration: 3.336784601s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:28:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.336804094s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 1.654388525s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:27:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.65442496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..........X............................X..................] duration: 3.240850009s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:27:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.240898126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 1.668763364s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:26:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.668805252s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 1.646788842s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:26:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.646830677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..........X............................X.................] duration: 3.308605004s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:26:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.308631365s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 1.844876101s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:25:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.844910092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..........X............................X................] duration: 3.765031565s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:25:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.765064597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..........X............................X...............] duration: 3.746916749s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:24:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.746951766s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001555288s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:24:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001649497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..........X............................X..............] duration: 3.35084168s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:23:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.350872582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..........X............................X.............] duration: 3.610580784s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:22:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.610613291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..........X............................X............] duration: 3.407385651s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:21:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.407421173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..........X............................X...........] duration: 3.940629072s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:20:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.940648712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..........X............................X..........] duration: 3.250217964s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:19:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.250253663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..........X............................X.........] duration: 4.864630973s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:18:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.864675177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X..........X............................X........] duration: 3.307444997s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:17:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.307545248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..........X............................X.......] duration: 4.433914713s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:17:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.433945693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..........X............................X......] duration: 3.075465735s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:16:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.075530819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X..........X............................X.....] duration: 3.572857152s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:16:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.5728907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X..........X............................X....] duration: 3.65218587s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:15:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.652215462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X..........X............................X...] duration: 3.277060365s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:14:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.277102153s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X................X..........X............................X..] duration: 3.428475889s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:13:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.428501258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X................X..........X............................X.] duration: 3.368720689s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:12:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.368745299s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X................X..........X............................X] duration: 1m0.109800483s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:12:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.109828772s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X................X..........X............................] duration: 4.287314759s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:11:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.287360322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X................X..........X...........................] duration: 3.500283169s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:10:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.500315134s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X................X..........X..........................] duration: 4.124271335s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:09:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.124327106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X................X..........X.........................] duration: 4.313779755s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:08:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.313883229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X................X..........X........................] duration: 3.528939566s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:06:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.52897321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.......X................X..........X.......................] duration: 3.560389906s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:06:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.560448648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX.......X................X..........X......................] duration: 3.261048383s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:05:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.26108931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX.......X................X..........X.....................] duration: 5.888154475s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:04:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.888204586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX.......X................X..........X....................] duration: 4.19138375s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:03:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.191432995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX.......X................X..........X...................] duration: 5.277039435s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:03:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.27708981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX.......X................X..........X..................] duration: 4.372431337s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:02:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.372463143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XX.......X................X..........X.................] duration: 3.376546171s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:02:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.37657943s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)