Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXX.XXX.XXXXX.XXX.XXX.....X........XX.......X...............] duration: 3.199424417s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:37:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.199528796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [XXX.X.XX..XX.X.....XX..X....................................] duration: 1.673249435s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:37:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.673341402s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.00564815s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:37:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.005684541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X..............] duration: 4.096486841s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:37:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.096522909s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [XXXX.X.XX..XX.X.....XX..X...................................] duration: 1.732483721s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:36:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.732513416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 3.098614608s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:36:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.098644451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X.............] duration: 7.229874568s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:36:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 7.229973049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.968956787s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:36:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.968985118s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XXXXX.X.XX..XX.X.....XX..X..................................] duration: 2.21158961s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:35:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.21162893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X............] duration: 10.768676347s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:35:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 10.768712276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X...........] duration: 9.096868865s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:35:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 9.096897383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 3.142042288s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:35:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.142076391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XXXXXX.X.XX..XX.X.....XX..X.................................] duration: 1.717925374s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:34:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.717974601s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 3.234759544s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:34:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.234789358s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [XXXXXXX.X.XX..XX.X.....XX..X................................] duration: 2.213264206s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:34:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.213298058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X..........] duration: 6.365411256s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:34:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.365528651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 3.557072545s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:33:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.557104141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X.........] duration: 8.623718243s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:33:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 8.623760292s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [XXXXXXXX.X.XX..XX.X.....XX..X...............................] duration: 1.737523974s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:33:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.737555195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 3.676462123s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:33:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.676492108s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 3.658665606s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:33:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.658708386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [XXXXXXXXX.X.XX..XX.X.....XX..X..............................] duration: 1.814518836s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:32:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.815303192s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 3.106452685s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:32:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.10652641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X........] duration: 5.26837415s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:32:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.268414117s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X..........X................................................] duration: 3.660820417s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:32:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.660851757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X.......] duration: 18.713859569s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:31:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 18.713975539s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XXXXXXXXXX.X.XX..XX.X.....XX..X.............................] duration: 1.784301356s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:31:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.784331609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X..........X...............................................] duration: 3.039090931s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:31:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.039125924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X......] duration: 5.103259476s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:31:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.103298365s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [XXXXXXXXXXX.X.XX..XX.X.....XX..X............................] duration: 2.203657105s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:30:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.203685484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X..........X..............................................] duration: 3.644700364s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:30:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.644790299s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X.....] duration: 3.647224453s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:30:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.64729487s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.X.XX..XX.X.....XX..X...........................] duration: 2.273235487s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:29:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.273262019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X....] duration: 6.692178424s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:29:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.692310424s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X..........X.............................................] duration: 3.054395754s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:29:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.054436437s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X...] duration: 4.833905251s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:28:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.833926627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.X.XX..XX.X.....XX..X..........................] duration: 2.310968379s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:28:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.310997385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.X.XX..XX.X.....XX..X.........................] duration: 1.796199815s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:28:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.796217032s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X..........X............................................] duration: 3.26712144s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:28:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.267163535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 34 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X..] duration: 3.725460545s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:27:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.725604699s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.X.XX..XX.X.....XX..X........................] duration: 1.707590752s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:27:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.707629346s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X....X..........X...........................................] duration: 3.101098366s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:27:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.101125097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 35 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X.] duration: 4.186442795s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:26:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.186464578s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......X] duration: 1m0.103521763s trace: https://jaeger.dev.storj.io/trace/7a32556c3299f77d FAIL (2026-04-18 02:26:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pload=240474]: Done begin upload segment. Apr 18 02:25:51.051303397 (250h9m1.645414643s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052559793 (250h9m1.646671043s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052600155 (250h9m1.646711385s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.052639728 (250h9m1.646750954s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053090817 (250h9m1.647202066s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053231933 (250h9m1.647343178s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053514417 (250h9m1.647625667s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053720653 (250h9m1.647831902s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.053903088 (250h9m1.648014338s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054285730 (250h9m1.648396983s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054397651 (250h9m1.648508896s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054802296 (250h9m1.648913546s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.054824840 (250h9m1.648936086s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055258766 (250h9m1.649370033s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055330406 (250h9m1.649441657s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055712191 (250h9m1.649823437s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.055870389 (250h9m1.649981634s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.056357114 (250h9m1.650468364s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Uploading piece... Apr 18 02:25:51.142743972 (250h9m1.736855217s) [noise=true , address=145.79.227.38:4998 , piece_id=OAQUJLFO36DXMTQTPTHQGENTNSH322K5AR6OUNCVPP7BNLG5L4SA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144049380 (250h9m1.738160628s) [noise=true , address=145.79.227.31:4818 , piece_id=COD6Y2NRXL4C4K33H6SXOWAKOOUFHEZWWGAXYTOCZCZEJDQDFROQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.144916153 (250h9m1.739027412s) [noise=true , address=145.79.227.49:4355 , piece_id=L7DYM5TPFNKMXFQQD5EEWHOEUPZZA32LG7MM4YVENHZDI3JM62WQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:145.79.227.49:4355): protocol: rpc: dial tcp 145.79.227.49:4355: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:25:51.632081968 (250h9m2.226193222s) [upload=240474]: Next returned nil segment Apr 18 02:25:51.632102736 (250h9m2.226213986s) [upload=240474]: Waiting for error group managing segments... Apr 18 02:25:51.779193064 (250h9m2.373304314s) [noise=true , address=145.79.227.13:4818 , piece_id=HKLCFYE35YBNVHQRCJD3HKP62TYO3A75I7E4MT2DZLPI4UFCY2DQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.166785373 (250h9m2.760896615s) [noise=true , address=167.17.72.70:4752 , piece_id=2FRYMES3YSXNKWCXX7RIHKHOJITURTPV36JSHMFLBEDMG6ECLZKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.184676686 (250h9m2.778787936s) [noise=true , address=167.17.72.68:4643 , piece_id=SPR27ZSBXU4ZYUQSH5NDUXLHVWRYEDXN5OEXWCLVCNX62VKSSRPA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.207075381 (250h9m2.801186622s) [noise=true , address=167.17.72.71:4791 , piece_id=FE65FAXNTOQVDNZBK7IVRUVFHTTHC7PLLEKRPEGI5S3QEGVWCSHQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.299059914 (250h9m2.893171157s) [noise=true , address=193.149.164.71:4290 , piece_id=V37K6CSZYGYH2ME5ZZ4B2HG6MLUCLZGZD24XKHMTYRM36QKZSG7A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.302165775 (250h9m2.89627702s) [noise=true , address=193.149.164.66:4988 , piece_id=Q6SQLDYRR3UFVRHPPXENV6QYRT6DE7YRHWNFODJRYKSUQAQKA77A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.465614425 (250h9m3.059725667s) [noise=true , address=79.127.205.234:4202 , piece_id=RAQ6ZJQGAX54WPBH4L2YCKVPOGNJIIVGC4Y4DMUSEGLVVIMM72RA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.479819483 (250h9m3.073930722s) [noise=true , address=79.127.205.233:4175 , piece_id=YI6X3CSWPNJGRCDMN4IRVSKBHBY5ATPGRDKKARVTR6DEOAJT5AQA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.533279958 (250h9m3.127391199s) [noise=true , address=79.127.205.237:4228 , piece_id=7Y5TZ4EGLASY4Q3RULNCXJZXCRLACSB4LHLW3NQIBMTPXNQI7E6Q, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.562759587 (250h9m3.156870833s) [noise=true , address=79.127.205.235:4257 , piece_id=L3UYPZLCQB3M2XX5UDQCB4BO7FBHFUJPRH6HCVXSORMUMHARLLKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:52.742002496 (250h9m3.336113738s) [noise=true , address=145.79.227.29:4961 , piece_id=5PLSCMIMF47SC5FTO56TFL7DTX5RYSWIXDHB4CWPXGVYBVML4MRQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: <nil> Apr 18 02:25:53.461038287 (250h9m4.055149536s) [noise=true , address=145.79.227.11:4743 , piece_id=2E66S2TDFWJQ5I77W3O4O7QKBAXFERC4CK3WYJHWWSJEKFXOXN4A, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:145.79.227.11:4743): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/b8/log-00000000000227b8-00005052: no space left on device; EOF Apr 18 02:26:50.666039093 (250h10m1.260150343s) [noise=true , address=145.79.227.33:4885 , piece_id=RJFDD7ZDNMX3QC26DQ2P4ET5ZXQVWWZCWJ2RQATYZ4UJGH3LYXKA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666101955 (250h10m1.260213203s) [noise=true , address=145.79.227.44:4220 , piece_id=7HSLW3EM6F75GFT62K5ZEV4R2RXCNOCDJRFLMW4JRY5EVUPWL2KQ, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:145.79.227.44:4220): protocol: rpc: dial tcp 145.79.227.44:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666324349 (250h10m1.260435599s) [noise=true , address=167.17.72.69:4734 , piece_id=ZR2CLBAPDR3WNVD7CYOJTN7J4LBG7HQX4CAGHK5LL2RYA7WKGMCA, seg_pos={0 0}, upload=240474]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:55810->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:26:50.666367389 (250h10m1.260478826s) [seg_pos={0 0}, upload=240474]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:26:50.767707041 (250h10m1.361818273s) [upload=240474]: Done uploading segments... Apr 18 02:26:50.767750393 (250h10m1.361861649s) [upload=240474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.103546773s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.X.XX..XX.X.....XX..X.......................] duration: 2.268692935s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:26:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.268726503s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [XX....X..........X..........................................] duration: 2.965158453s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:26:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.96518776s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.X.XX..XX.X.....XX..X......................] duration: 1.68815667s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:25:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.688193337s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [XXX....X..........X.........................................] duration: 2.989530797s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:25:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.989562785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXX.XXX.....X........XX.......] duration: 8.722867879s trace: https://jaeger.dev.storj.io/trace/78e9ec05d97d6338 FAIL (2026-04-18 02:25:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 218041s) [upload=240394]: Uploading segments... Apr 18 02:20:32.668377733 (250h3m43.262488977s) [upload=240394]: Got next segment. Inline: false Apr 18 02:20:32.668531312 (250h3m43.262642556s) [upload=240394]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:20:32.668538982 (250h3m43.262650231s) [upload=240394]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:20:32.798501929 (250h3m43.392613172s) [seg_pos={0 0}, upload=240394]: Begin upload segment... Apr 18 02:20:32.798619280 (250h3m43.392730526s) [noise=true , address=145.79.227.44:4210 , piece_id=YBG7SBXSTSZ76Z6KKNWA3KVIAZC2SUWR7J23WYXNKW3ZG6IC7YUA, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.799770801 (250h3m43.393882048s) [noise=true , address=145.79.227.34:4936 , piece_id=IMAWTM7W7FSUGL7W6A3DMDXGBVSQWRETLIUYVPAZBZZO3PLD5GSQ, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.801315980 (250h3m43.395427225s) [noise=true , address=167.17.72.70:4750 , piece_id=LDY7KHBEAITT2PPIXF3HTMX7P35GAUHTJ3GKVJ4HFEFNAYWAMCEA, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.802080311 (250h3m43.39619156s) [seg_pos={0 0}, upload=240394]: Done begin upload segment. Apr 18 02:20:32.802155107 (250h3m43.396266358s) [noise=true , address=167.17.72.71:4814 , piece_id=HBFGG3KJLQDUAKTZIGAHKFIQQG57CIW25V2O62SIXNP5UONHVOZQ, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.802798648 (250h3m43.396909893s) [noise=true , address=145.79.227.21:4480 , piece_id=O6CJTOSH32EXOFM2RMLWOHXQWCWDC7O6MQXDA47AEHV3Z3PR6DKQ, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.805202083 (250h3m43.399313337s) [noise=true , address=145.79.227.22:4523 , piece_id=UTBMHUN7A6ANYHEDWL7LK2O475TURNNFQE5BP7KOGIX6S2DJVITA, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.806022799 (250h3m43.40013405s) [noise=true , address=193.149.164.71:4297 , piece_id=TI3KYOJEKKU52LTRJSTDTIL7Q3WOJT37OH7SNAU2SCOKARGVBV6A, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.806689954 (250h3m43.400801208s) [noise=true , address=79.127.205.234:4177 , piece_id=TBZKC4KF2GNTVIRE6EW4V4BRSCVTWATGF2AUSWOIFV5LIJOMCJUQ, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.807333559 (250h3m43.401444809s) [noise=true , address=64.49.13.27:4964 , piece_id=TD6XIY7SC57JZMKI4NIYMPQ7CVI3YVFP7DG5WSMWDV2CS45YZ3UA, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.807960573 (250h3m43.402071824s) [noise=true , address=145.79.227.33:4898 , piece_id=GZLJXPWCKPUPN2DT4XMKJP7NCOAMLHOZFHKPHXZXWUGSDUR3I5BA, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.808629541 (250h3m43.402740792s) [noise=true , address=145.79.227.13:4819 , piece_id=YZQI6Y5XCBAG67PJU2RRXOUNHL337OQTVSZQFXQMONB5N2BKPBMQ, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.811002620 (250h3m43.405113872s) [noise=true , address=77.90.31.10:4546 , piece_id=YGFACWLAWPV77U64BF66UGZ6BDGI3ZBGAMZ7UWIVPZL75H4BHWAQ, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.813996142 (250h3m43.408107393s) [noise=true , address=167.17.72.68:4640 , piece_id=XRRX6NCKZOGDH4DRIFQRJVXWJLKKP4EMY3FH7Q6CJK2ET2I5DMJA, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.815672703 (250h3m43.409783956s) [noise=true , address=79.127.205.235:4263 , piece_id=CYYAHCUSLBHF57GUH3ONAMWAVJ6GI42PD4GA4V645U2HGW7QNDUQ, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.817126477 (250h3m43.411237731s) [noise=true , address=77.90.31.17:4777 , piece_id=AMQKC26XVEW6ZU7ZQ66M6JZ3YTMUZSVRFZ6AFXZ5NJJ7ZPW2YELA, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.817333127 (250h3m43.411444377s) [noise=true , address=79.127.205.233:4151 , piece_id=QMUWF2A25PTGCD3MSRBYUWNF4VIYE34AZ4J64LJIZQ44PTZDFXVQ, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.819369687 (250h3m43.413480988s) [noise=true , address=145.79.227.49:4367 , piece_id=TAJFZHQTZOYH37C4VKAYY6RJOMEOFM3HVHSCIHAWBFSFHHGJYHNA, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.820113899 (250h3m43.41422515s) [noise=true , address=145.79.227.51:4424 , piece_id=JWTXMHLA65FORFSUA66LLNMD4FOYVLR7DKYSWAETZLHBWWGV3ZFA, seg_pos={0 0}, upload=240394]: Uploading piece... Apr 18 02:20:32.901705476 (250h3m43.495816721s) [noise=true , address=145.79.227.33:4898 , piece_id=GZLJXPWCKPUPN2DT4XMKJP7NCOAMLHOZFHKPHXZXWUGSDUR3I5BA, seg_pos={0 0}, upload=240394]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:145.79.227.33:4898): protocol: rpc: dial tcp 145.79.227.33:4898: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:20:32.910678821 (250h3m43.504790073s) [noise=true , address=145.79.227.49:4367 , piece_id=TAJFZHQTZOYH37C4VKAYY6RJOMEOFM3HVHSCIHAWBFSFHHGJYHNA, seg_pos={0 0}, upload=240394]: Done uploading piece. err: ecclient: upload failed (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK, address:145.79.227.49:4367): protocol: rpc: dial tcp 145.79.227.49:4367: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:20:32.961158335 (250h3m43.555269584s) [noise=true , address=167.17.72.70:4750 , piece_id=LDY7KHBEAITT2PPIXF3HTMX7P35GAUHTJ3GKVJ4HFEFNAYWAMCEA, seg_pos={0 0}, upload=240394]: Done uploading piece. err: ecclient: upload failed (node:16spDGswJh7AS4m7xYwkJF7p8hARtJdR2hmYw1aWz9NrZDM1jj, address:167.17.72.70:4750): protocol: write tcp 139.59.144.22:50904->167.17.72.70:4750: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:20:32.968691165 (250h3m43.562802431s) [noise=true , address=167.17.72.68:4640 , piece_id=XRRX6NCKZOGDH4DRIFQRJVXWJLKKP4EMY3FH7Q6CJK2ET2I5DMJA, seg_pos={0 0}, upload=240394]: Done uploading piece. err: ecclient: upload failed (node:1X1An1SwLyzf72pHcN7pYsuVchBWbVWJvbNSm8v3cFcgwzS3P1, address:167.17.72.68:4640): protocol: write tcp 139.59.144.22:59100->167.17.72.68:4640: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:20:33.366076601 (250h3m43.960187861s) [upload=240394]: Next returned nil segment Apr 18 02:20:33.366100451 (250h3m43.960211708s) [upload=240394]: Waiting for error group managing segments... Apr 18 02:20:33.545413199 (250h3m44.13952444s) [noise=true , address=145.79.227.21:4480 , piece_id=O6CJTOSH32EXOFM2RMLWOHXQWCWDC7O6MQXDA47AEHV3Z3PR6DKQ, seg_pos={0 0}, upload=240394]: Done uploading piece. err: <nil> Apr 18 02:20:33.557342012 (250h3m44.151453262s) [noise=true , address=145.79.227.13:4819 , piece_id=YZQI6Y5XCBAG67PJU2RRXOUNHL337OQTVSZQFXQMONB5N2BKPBMQ, seg_pos={0 0}, upload=240394]: Done uploading piece. err: <nil> Apr 18 02:20:33.966221849 (250h3m44.560333098s) [noise=true , address=167.17.72.71:4814 , piece_id=HBFGG3KJLQDUAKTZIGAHKFIQQG57CIW25V2O62SIXNP5UONHVOZQ, seg_pos={0 0}, upload=240394]: Done uploading piece. err: <nil> Apr 18 02:20:34.028680484 (250h3m44.622791735s) [noise=true , address=145.79.227.51:4424 , piece_id=JWTXMHLA65FORFSUA66LLNMD4FOYVLR7DKYSWAETZLHBWWGV3ZFA, seg_pos={0 0}, upload=240394]: Done uploading piece. err: <nil> Apr 18 02:20:34.087094362 (250h3m44.681205588s) [noise=true , address=64.49.13.27:4964 , piece_id=TD6XIY7SC57JZMKI4NIYMPQ7CVI3YVFP7DG5WSMWDV2CS45YZ3UA, seg_pos={0 0}, upload=240394]: Done uploading piece. err: <nil> Apr 18 02:20:34.096110354 (250h3m44.690221598s) [noise=true , address=193.149.164.71:4297 , piece_id=TI3KYOJEKKU52LTRJSTDTIL7Q3WOJT37OH7SNAU2SCOKARGVBV6A, seg_pos={0 0}, upload=240394]: Done uploading piece. err: <nil> Apr 18 02:20:34.142002427 (250h3m44.736113668s) [noise=true , address=79.127.205.235:4263 , piece_id=CYYAHCUSLBHF57GUH3ONAMWAVJ6GI42PD4GA4V645U2HGW7QNDUQ, seg_pos={0 0}, upload=240394]: Done uploading piece. err: <nil> Apr 18 02:20:34.206517179 (250h3m44.800628423s) [noise=true , address=79.127.205.234:4177 , piece_id=TBZKC4KF2GNTVIRE6EW4V4BRSCVTWATGF2AUSWOIFV5LIJOMCJUQ, seg_pos={0 0}, upload=240394]: Done uploading piece. err: <nil> Apr 18 02:20:34.215638225 (250h3m44.809749466s) [noise=true , address=79.127.205.233:4151 , piece_id=QMUWF2A25PTGCD3MSRBYUWNF4VIYE34AZ4J64LJIZQ44PTZDFXVQ, seg_pos={0 0}, upload=240394]: Done uploading piece. err: <nil> Apr 18 02:20:34.605492108 (250h3m45.199603338s) [noise=true , address=145.79.227.22:4523 , piece_id=UTBMHUN7A6ANYHEDWL7LK2O475TURNNFQE5BP7KOGIX6S2DJVITA, seg_pos={0 0}, upload=240394]: Done uploading piece. err: <nil> Apr 18 02:20:34.822934405 (250h3m45.417045669s) [noise=true , address=77.90.31.10:4546 , piece_id=YGFACWLAWPV77U64BF66UGZ6BDGI3ZBGAMZ7UWIVPZL75H4BHWAQ, seg_pos={0 0}, upload=240394]: Done uploading piece. err: <nil> Apr 18 02:20:36.842912495 (250h3m47.437023763s) [noise=true , address=77.90.31.17:4777 , piece_id=AMQKC26XVEW6ZU7ZQ66M6JZ3YTMUZSVRFZ6AFXZ5NJJ7ZPW2YELA, seg_pos={0 0}, upload=240394]: Done uploading piece. err: <nil> Apr 18 02:20:43.279384547 (250h3m53.873495795s) [noise=true , address=145.79.227.34:4936 , piece_id=IMAWTM7W7FSUGL7W6A3DMDXGBVSQWRETLIUYVPAZBZZO3PLD5GSQ, seg_pos={0 0}, upload=240394]: Done uploading piece. err: <nil> Apr 18 02:21:32.358582612 (250h4m42.95269386s) [noise=true , address=145.79.227.44:4210 , piece_id=YBG7SBXSTSZ76Z6KKNWA3KVIAZC2SUWR7J23WYXNKW3ZG6IC7YUA, seg_pos={0 0}, upload=240394]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:145.79.227.44:4210): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:21:32.358646595 (250h4m42.952757827s) [seg_pos={0 0}, upload=240394]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:21:32.464561894 (250h4m43.058673133s) [upload=240394]: Done uploading segments... Apr 18 02:21:32.464598829 (250h4m43.05871007s) [upload=240394]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.722902412s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.X.XX..XX.X.....XX..X.....................] duration: 1.668006534s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:25:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.668034089s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)