Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.XX..XX.X.....XX..X.........................................] duration: 1.670719684s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:41:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.670745881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX.XXXXX.XXX.XXX.....X........XX.......X................X...] duration: 3.537828641s trace: https://jaeger.dev.storj.io/trace/31005e07d94faebc FAIL (2026-04-18 02:41:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 02:39:18.903757261 (250h22m29.4978685s) [upload=240701]: Got next segment. Inline: false Apr 18 02:39:18.903830365 (250h22m29.497941603s) [upload=240701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:39:18.903837205 (250h22m29.497948444s) [upload=240701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:39:19.014891342 (250h22m29.609002572s) [seg_pos={0 0}, upload=240701]: Begin upload segment... Apr 18 02:39:19.014972185 (250h22m29.609083415s) [seg_pos={0 0}, upload=240701]: Done begin upload segment. Apr 18 02:39:19.015019360 (250h22m29.609130603s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015032108 (250h22m29.609143353s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015104542 (250h22m29.609215781s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015691354 (250h22m29.609802603s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.016299539 (250h22m29.610410788s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017036654 (250h22m29.611147898s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017269327 (250h22m29.611380573s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017746091 (250h22m29.611857335s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018291629 (250h22m29.612402878s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018353743 (250h22m29.612464986s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018721265 (250h22m29.612832515s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018839235 (250h22m29.61295048s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019288928 (250h22m29.613400176s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019746809 (250h22m29.613858054s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020163463 (250h22m29.614274708s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020821300 (250h22m29.614932548s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021249261 (250h22m29.615360503s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021618436 (250h22m29.615729677s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.164125748 (250h22m29.758236994s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:60394->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.200099142 (250h22m29.79421039s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34470->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.580941409 (250h22m30.175052661s) [upload=240701]: Next returned nil segment Apr 18 02:39:19.580961207 (250h22m30.175072451s) [upload=240701]: Waiting for error group managing segments... Apr 18 02:39:19.741177031 (250h22m30.33528829s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.745737016 (250h22m30.339848276s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.789115076 (250h22m30.383226333s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.848298928 (250h22m30.442410169s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.199475147 (250h22m30.793586388s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.319894286 (250h22m30.914005536s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.356429032 (250h22m30.950540277s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.383013757 (250h22m30.977125007s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.384162295 (250h22m30.978273543s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.424713162 (250h22m31.018824409s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.439978786 (250h22m31.034090027s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.476434966 (250h22m31.070546206s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.802472404 (250h22m31.396583644s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:40:18.617604639 (250h23m29.211715882s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:53148->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.617816606 (250h23m29.211927855s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.618072575 (250h23m29.212183825s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:145.79.227.49:4363): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:40:18.618108549 (250h23m29.212219783s) [seg_pos={0 0}, upload=240701]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:40:18.722218529 (250h23m29.31632977s) [upload=240701]: Done uploading segments... Apr 18 02:40:18.722256343 (250h23m29.316367586s) [upload=240701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.537853856s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X.XX..XX.X.....XX..X........................................] duration: 2.177890384s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:40:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.177924816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXX.XXXXX.XXX.XXX.....X........XX.......X................X..] duration: 3.711085666s trace: https://jaeger.dev.storj.io/trace/31005e07d94faebc FAIL (2026-04-18 02:40:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 02:39:18.903757261 (250h22m29.4978685s) [upload=240701]: Got next segment. Inline: false Apr 18 02:39:18.903830365 (250h22m29.497941603s) [upload=240701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:39:18.903837205 (250h22m29.497948444s) [upload=240701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:39:19.014891342 (250h22m29.609002572s) [seg_pos={0 0}, upload=240701]: Begin upload segment... Apr 18 02:39:19.014972185 (250h22m29.609083415s) [seg_pos={0 0}, upload=240701]: Done begin upload segment. Apr 18 02:39:19.015019360 (250h22m29.609130603s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015032108 (250h22m29.609143353s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015104542 (250h22m29.609215781s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015691354 (250h22m29.609802603s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.016299539 (250h22m29.610410788s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017036654 (250h22m29.611147898s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017269327 (250h22m29.611380573s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017746091 (250h22m29.611857335s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018291629 (250h22m29.612402878s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018353743 (250h22m29.612464986s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018721265 (250h22m29.612832515s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018839235 (250h22m29.61295048s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019288928 (250h22m29.613400176s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019746809 (250h22m29.613858054s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020163463 (250h22m29.614274708s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020821300 (250h22m29.614932548s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021249261 (250h22m29.615360503s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021618436 (250h22m29.615729677s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.164125748 (250h22m29.758236994s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:60394->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.200099142 (250h22m29.79421039s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34470->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.580941409 (250h22m30.175052661s) [upload=240701]: Next returned nil segment Apr 18 02:39:19.580961207 (250h22m30.175072451s) [upload=240701]: Waiting for error group managing segments... Apr 18 02:39:19.741177031 (250h22m30.33528829s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.745737016 (250h22m30.339848276s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.789115076 (250h22m30.383226333s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.848298928 (250h22m30.442410169s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.199475147 (250h22m30.793586388s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.319894286 (250h22m30.914005536s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.356429032 (250h22m30.950540277s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.383013757 (250h22m30.977125007s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.384162295 (250h22m30.978273543s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.424713162 (250h22m31.018824409s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.439978786 (250h22m31.034090027s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.476434966 (250h22m31.070546206s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.802472404 (250h22m31.396583644s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:40:18.617604639 (250h23m29.211715882s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:53148->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.617816606 (250h23m29.211927855s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.618072575 (250h23m29.212183825s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:145.79.227.49:4363): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:40:18.618108549 (250h23m29.212219783s) [seg_pos={0 0}, upload=240701]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:40:18.722218529 (250h23m29.31632977s) [upload=240701]: Done uploading segments... Apr 18 02:40:18.722256343 (250h23m29.316367586s) [upload=240701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.711131583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XXX.XXXXX.XXX.XXX.....X........XX.......X................X.] duration: 3.309541632s trace: https://jaeger.dev.storj.io/trace/31005e07d94faebc FAIL (2026-04-18 02:40:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 02:39:18.903757261 (250h22m29.4978685s) [upload=240701]: Got next segment. Inline: false Apr 18 02:39:18.903830365 (250h22m29.497941603s) [upload=240701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:39:18.903837205 (250h22m29.497948444s) [upload=240701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:39:19.014891342 (250h22m29.609002572s) [seg_pos={0 0}, upload=240701]: Begin upload segment... Apr 18 02:39:19.014972185 (250h22m29.609083415s) [seg_pos={0 0}, upload=240701]: Done begin upload segment. Apr 18 02:39:19.015019360 (250h22m29.609130603s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015032108 (250h22m29.609143353s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015104542 (250h22m29.609215781s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015691354 (250h22m29.609802603s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.016299539 (250h22m29.610410788s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017036654 (250h22m29.611147898s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017269327 (250h22m29.611380573s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017746091 (250h22m29.611857335s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018291629 (250h22m29.612402878s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018353743 (250h22m29.612464986s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018721265 (250h22m29.612832515s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018839235 (250h22m29.61295048s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019288928 (250h22m29.613400176s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019746809 (250h22m29.613858054s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020163463 (250h22m29.614274708s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020821300 (250h22m29.614932548s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021249261 (250h22m29.615360503s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021618436 (250h22m29.615729677s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.164125748 (250h22m29.758236994s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:60394->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.200099142 (250h22m29.79421039s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34470->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.580941409 (250h22m30.175052661s) [upload=240701]: Next returned nil segment Apr 18 02:39:19.580961207 (250h22m30.175072451s) [upload=240701]: Waiting for error group managing segments... Apr 18 02:39:19.741177031 (250h22m30.33528829s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.745737016 (250h22m30.339848276s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.789115076 (250h22m30.383226333s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.848298928 (250h22m30.442410169s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.199475147 (250h22m30.793586388s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.319894286 (250h22m30.914005536s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.356429032 (250h22m30.950540277s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.383013757 (250h22m30.977125007s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.384162295 (250h22m30.978273543s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.424713162 (250h22m31.018824409s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.439978786 (250h22m31.034090027s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.476434966 (250h22m31.070546206s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.802472404 (250h22m31.396583644s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:40:18.617604639 (250h23m29.211715882s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:53148->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.617816606 (250h23m29.211927855s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.618072575 (250h23m29.212183825s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:145.79.227.49:4363): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:40:18.618108549 (250h23m29.212219783s) [seg_pos={0 0}, upload=240701]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:40:18.722218529 (250h23m29.31632977s) [upload=240701]: Done uploading segments... Apr 18 02:40:18.722256343 (250h23m29.316367586s) [upload=240701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.309565482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.XXX.XXXXX.XXX.XXX.....X........XX.......X................X] duration: 1m0.107479288s trace: https://jaeger.dev.storj.io/trace/31005e07d94faebc FAIL (2026-04-18 02:40:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 02:39:18.903757261 (250h22m29.4978685s) [upload=240701]: Got next segment. Inline: false Apr 18 02:39:18.903830365 (250h22m29.497941603s) [upload=240701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:39:18.903837205 (250h22m29.497948444s) [upload=240701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:39:19.014891342 (250h22m29.609002572s) [seg_pos={0 0}, upload=240701]: Begin upload segment... Apr 18 02:39:19.014972185 (250h22m29.609083415s) [seg_pos={0 0}, upload=240701]: Done begin upload segment. Apr 18 02:39:19.015019360 (250h22m29.609130603s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015032108 (250h22m29.609143353s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015104542 (250h22m29.609215781s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015691354 (250h22m29.609802603s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.016299539 (250h22m29.610410788s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017036654 (250h22m29.611147898s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017269327 (250h22m29.611380573s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017746091 (250h22m29.611857335s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018291629 (250h22m29.612402878s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018353743 (250h22m29.612464986s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018721265 (250h22m29.612832515s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018839235 (250h22m29.61295048s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019288928 (250h22m29.613400176s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019746809 (250h22m29.613858054s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020163463 (250h22m29.614274708s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020821300 (250h22m29.614932548s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021249261 (250h22m29.615360503s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021618436 (250h22m29.615729677s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.164125748 (250h22m29.758236994s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:60394->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.200099142 (250h22m29.79421039s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34470->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.580941409 (250h22m30.175052661s) [upload=240701]: Next returned nil segment Apr 18 02:39:19.580961207 (250h22m30.175072451s) [upload=240701]: Waiting for error group managing segments... Apr 18 02:39:19.741177031 (250h22m30.33528829s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.745737016 (250h22m30.339848276s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.789115076 (250h22m30.383226333s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.848298928 (250h22m30.442410169s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.199475147 (250h22m30.793586388s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.319894286 (250h22m30.914005536s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.356429032 (250h22m30.950540277s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.383013757 (250h22m30.977125007s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.384162295 (250h22m30.978273543s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.424713162 (250h22m31.018824409s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.439978786 (250h22m31.034090027s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.476434966 (250h22m31.070546206s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.802472404 (250h22m31.396583644s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:40:18.617604639 (250h23m29.211715882s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:53148->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.617816606 (250h23m29.211927855s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.618072575 (250h23m29.212183825s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:145.79.227.49:4363): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:40:18.618108549 (250h23m29.212219783s) [seg_pos={0 0}, upload=240701]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:40:18.722218529 (250h23m29.31632977s) [upload=240701]: Done uploading segments... Apr 18 02:40:18.722256343 (250h23m29.316367586s) [upload=240701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.107503132s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X.XX..XX.X.....XX..X.......................................] duration: 1.70043881s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:40:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.700476901s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.740251233s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:39:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.740327033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X.X.XX..XX.X.....XX..X......................................] duration: 1.640659219s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:39:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.640695548s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX.XXX.XXXXX.XXX.XXX.....X........XX.......X................] duration: 3.292670344s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:38:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.292710562s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [XX.X.XX..XX.X.....XX..X.....................................] duration: 1.651286519s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:38:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.651318318s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.006942067s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:38:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.007039005s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXX.XXX.XXXXX.XXX.XXX.....X........XX.......X...............] duration: 3.199424417s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:37:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.199528796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [XXX.X.XX..XX.X.....XX..X....................................] duration: 1.673249435s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:37:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.673341402s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.00564815s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:37:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.005684541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X..............] duration: 4.096486841s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:37:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.096522909s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [XXXX.X.XX..XX.X.....XX..X...................................] duration: 1.732483721s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:36:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.732513416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 3.098614608s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:36:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.098644451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X.............] duration: 7.229874568s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:36:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 7.229973049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.968956787s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:36:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.968985118s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XXXXX.X.XX..XX.X.....XX..X..................................] duration: 2.21158961s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:35:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.21162893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X............] duration: 10.768676347s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:35:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 10.768712276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X...........] duration: 9.096868865s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:35:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 9.096897383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 3.142042288s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:35:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.142076391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XXXXXX.X.XX..XX.X.....XX..X.................................] duration: 1.717925374s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:34:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.717974601s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 3.234759544s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:34:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.234789358s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [XXXXXXX.X.XX..XX.X.....XX..X................................] duration: 2.213264206s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:34:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.213298058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X..........] duration: 6.365411256s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:34:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.365528651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 3.557072545s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:33:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.557104141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X.........] duration: 8.623718243s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:33:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 8.623760292s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [XXXXXXXX.X.XX..XX.X.....XX..X...............................] duration: 1.737523974s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:33:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.737555195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 3.676462123s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:33:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.676492108s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 3.658665606s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:33:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.658708386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [XXXXXXXXX.X.XX..XX.X.....XX..X..............................] duration: 1.814518836s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:32:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.815303192s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 3.106452685s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:32:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.10652641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X........] duration: 5.26837415s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:32:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.268414117s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X..........X................................................] duration: 3.660820417s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:32:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.660851757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X.......] duration: 18.713859569s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:31:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 18.713975539s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XXXXXXXXXX.X.XX..XX.X.....XX..X.............................] duration: 1.784301356s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:31:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.784331609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X..........X...............................................] duration: 3.039090931s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:31:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.039125924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X......] duration: 5.103259476s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:31:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.103298365s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [XXXXXXXXXXX.X.XX..XX.X.....XX..X............................] duration: 2.203657105s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:30:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.203685484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X..........X..............................................] duration: 3.644700364s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:30:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.644790299s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X.....] duration: 3.647224453s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:30:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.64729487s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.X.XX..XX.X.....XX..X...........................] duration: 2.273235487s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:29:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.273262019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X....] duration: 6.692178424s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:29:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.692310424s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X..........X.............................................] duration: 3.054395754s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:29:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.054436437s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X...] duration: 4.833905251s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:28:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.833926627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.X.XX..XX.X.....XX..X..........................] duration: 2.310968379s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:28:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.310997385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.X.XX..XX.X.....XX..X.........................] duration: 1.796199815s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:28:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.796217032s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)