Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXX.XXX] duration: 1m0.116189052s trace: https://jaeger.dev.storj.io/trace/2967fd859cac0203 FAIL (2026-04-18 02:08:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11.774780176s) [upload=240175]: Starting upload Apr 18 02:07:01.180757085 (249h50m11.774868329s) [upload=240175]: Done starting upload Apr 18 02:07:01.180985360 (249h50m11.77509659s) [upload=240175]: Uploading segments... Apr 18 02:07:01.181168233 (249h50m11.775279482s) [upload=240175]: Got next segment. Inline: false Apr 18 02:07:01.181243742 (249h50m11.775354995s) [upload=240175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:07:01.181252211 (249h50m11.775363439s) [upload=240175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:07:01.292933411 (249h50m11.887044657s) [seg_pos={0 0}, upload=240175]: Begin upload segment... Apr 18 02:07:01.293025664 (249h50m11.887136909s) [seg_pos={0 0}, upload=240175]: Done begin upload segment. Apr 18 02:07:01.293539607 (249h50m11.887650856s) [noise=true , address=79.127.205.235:4259 , piece_id=6VSSTNZ5MZE73ZYKH24FP3S6NMGPUOJ2YU7UVOFMALW7EMERL43A, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.294842538 (249h50m11.88895379s) [noise=true , address=167.17.72.70:4769 , piece_id=I53QO4KXRCYBX37QN3GO6ESWYQLLGRVJZGKYXNK37J2LVS5NWVGQ, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.295300817 (249h50m11.889412067s) [noise=true , address=193.149.164.143:4696 , piece_id=3DAAFESUBMPPN72P5TKVW75QG2AHSX4JV44MS3LTMTNP5YNPI7NQ, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.295682089 (249h50m11.889793333s) [noise=true , address=145.79.227.33:4879 , piece_id=DORIWIXSPB7SDRTZV52EJWJP2BI7MKWNX3O5NCY24EBTP6GBTTMA, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.296033265 (249h50m11.89014451s) [noise=true , address=77.90.31.11:4566 , piece_id=L36ZMC3ND56T6TSZFVHGS76KXIDJBLMEWWKUON3IKC2XXPIES4RQ, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.296397353 (249h50m11.890508604s) [noise=true , address=145.79.227.9:4399 , piece_id=XXOJ43XTJDKYIDBOEVOD44GR57LDHOP25EACTIS3RCWSAKZMZXYA, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.296744242 (249h50m11.89085549s) [noise=true , address=79.127.205.233:4153 , piece_id=XNSML5WZWBUDKTDWEUDDE35JOHAIGTFWIXN5UPMLX72LX5OUZTBA, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.297070106 (249h50m11.891181352s) [noise=true , address=145.79.227.11:4760 , piece_id=NQRE6BQXSFUTVEOZLT75O36UQPRBCBU5BDYHGHQDXXJQ5DYWTBEA, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.297406170 (249h50m11.891517417s) [noise=true , address=145.79.227.13:4820 , piece_id=WIVQAKIED5IGK2HMDDYTA5BX5S4I7ZX6LBHLFTQ4R3BWF3AWZGBA, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.297754626 (249h50m11.891865876s) [noise=true , address=193.149.164.71:4288 , piece_id=64NDF72ZNNJAF3O3XBBLOIKP7CNSIFUAPKV54MWCKAUJ3RJPG2RQ, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.298147982 (249h50m11.892259228s) [noise=true , address=167.17.72.71:4805 , piece_id=SP7LAF6GXX37RFT6U4M7IXLDAMUFISWUNHSKDJJ35O5ISMGDCDNA, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.298480553 (249h50m11.892591799s) [noise=true , address=167.17.72.68:4644 , piece_id=KDF4UKI57AAGYELLSKE4TCNJAAUALEWF2R2TAU6IULDG6RGBRXQQ, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.298842238 (249h50m11.89295348s) [noise=true , address=145.79.227.44:4205 , piece_id=M45SSYPWZRO6G67SFEDNSTYIOZC247SP7EUNQQYVA2TKA6DYWYJA, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.299206905 (249h50m11.893318155s) [noise=true , address=79.127.205.234:4201 , piece_id=3OX5R6T5GSGFXE2FZSCDTGY4UD3DKJA6IEPX5F2WHT6IK7NJOE7Q, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.299556039 (249h50m11.893667284s) [noise=true , address=145.79.227.21:4508 , piece_id=T2OCNTKI6OZCSNTKDXLNU2IBKOC4UAYCFFIWDYRSHPYGWLQUYFZA, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.299897686 (249h50m11.894008933s) [noise=true , address=167.17.72.69:4724 , piece_id=FP4XSTMRCELITDAFYCVMSCCD22BD3D4XYJGR2Z4QBQRWSXJDTD7Q, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.300248777 (249h50m11.894360028s) [noise=true , address=145.79.227.52:4452 , piece_id=CYAYISM2LSUW64UEO2M45UQL3AVW25UKDZXR4JLPVK2GK7PNESRA, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.305139115 (249h50m11.899250367s) [noise=true , address=145.79.227.38:4984 , piece_id=R5VAPPKIIPR7NIMTYOEHV6HNSZDWXSZRIJMF6DKLWMPKTSYQVI5A, seg_pos={0 0}, upload=240175]: Uploading piece... Apr 18 02:07:01.500436842 (249h50m12.094548092s) [noise=true , address=193.149.164.143:4696 , piece_id=3DAAFESUBMPPN72P5TKVW75QG2AHSX4JV44MS3LTMTNP5YNPI7NQ, seg_pos={0 0}, upload=240175]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34372->193.149.164.143:4696: write: connection refused; conn closed: op Apr 18 02:07:01.894721177 (249h50m12.48883242s) [upload=240175]: Next returned nil segment Apr 18 02:07:01.894741405 (249h50m12.488852661s) [upload=240175]: Waiting for error group managing segments... Apr 18 02:07:02.076974039 (249h50m12.671085286s) [noise=true , address=145.79.227.52:4452 , piece_id=CYAYISM2LSUW64UEO2M45UQL3AVW25UKDZXR4JLPVK2GK7PNESRA, seg_pos={0 0}, upload=240175]: Done uploading piece. err: <nil> Apr 18 02:07:02.130318046 (249h50m12.724429299s) [noise=true , address=145.79.227.13:4820 , piece_id=WIVQAKIED5IGK2HMDDYTA5BX5S4I7ZX6LBHLFTQ4R3BWF3AWZGBA, seg_pos={0 0}, upload=240175]: Done uploading piece. err: <nil> Apr 18 02:07:02.700143270 (249h50m13.294254517s) [noise=true , address=79.127.205.235:4259 , piece_id=6VSSTNZ5MZE73ZYKH24FP3S6NMGPUOJ2YU7UVOFMALW7EMERL43A, seg_pos={0 0}, upload=240175]: Done uploading piece. err: <nil> Apr 18 02:07:02.722235551 (249h50m13.31634681s) [noise=true , address=79.127.205.233:4153 , piece_id=XNSML5WZWBUDKTDWEUDDE35JOHAIGTFWIXN5UPMLX72LX5OUZTBA, seg_pos={0 0}, upload=240175]: Done uploading piece. err: <nil> Apr 18 02:07:02.729965985 (249h50m13.324077232s) [noise=true , address=79.127.205.234:4201 , piece_id=3OX5R6T5GSGFXE2FZSCDTGY4UD3DKJA6IEPX5F2WHT6IK7NJOE7Q, seg_pos={0 0}, upload=240175]: Done uploading piece. err: <nil> Apr 18 02:07:02.736816364 (249h50m13.330927615s) [noise=true , address=167.17.72.71:4805 , piece_id=SP7LAF6GXX37RFT6U4M7IXLDAMUFISWUNHSKDJJ35O5ISMGDCDNA, seg_pos={0 0}, upload=240175]: Done uploading piece. err: <nil> Apr 18 02:07:03.217725826 (249h50m13.811837072s) [noise=true , address=193.149.164.71:4288 , piece_id=64NDF72ZNNJAF3O3XBBLOIKP7CNSIFUAPKV54MWCKAUJ3RJPG2RQ, seg_pos={0 0}, upload=240175]: Done uploading piece. err: <nil> Apr 18 02:07:04.198688757 (249h50m14.792800008s) [noise=true , address=167.17.72.70:4769 , piece_id=I53QO4KXRCYBX37QN3GO6ESWYQLLGRVJZGKYXNK37J2LVS5NWVGQ, seg_pos={0 0}, upload=240175]: Done uploading piece. err: <nil> Apr 18 02:07:04.636978042 (249h50m15.231089404s) [noise=true , address=167.17.72.68:4644 , piece_id=KDF4UKI57AAGYELLSKE4TCNJAAUALEWF2R2TAU6IULDG6RGBRXQQ, seg_pos={0 0}, upload=240175]: Done uploading piece. err: <nil> Apr 18 02:07:06.138727885 (249h50m16.732839133s) [noise=true , address=145.79.227.9:4399 , piece_id=XXOJ43XTJDKYIDBOEVOD44GR57LDHOP25EACTIS3RCWSAKZMZXYA, seg_pos={0 0}, upload=240175]: Done uploading piece. err: <nil> Apr 18 02:07:06.553545637 (249h50m17.14765689s) [noise=true , address=77.90.31.11:4566 , piece_id=L36ZMC3ND56T6TSZFVHGS76KXIDJBLMEWWKUON3IKC2XXPIES4RQ, seg_pos={0 0}, upload=240175]: Done uploading piece. err: <nil> Apr 18 02:07:08.764696305 (249h50m19.358807547s) [noise=true , address=145.79.227.33:4879 , piece_id=DORIWIXSPB7SDRTZV52EJWJP2BI7MKWNX3O5NCY24EBTP6GBTTMA, seg_pos={0 0}, upload=240175]: Done uploading piece. err: <nil> Apr 18 02:07:09.639375880 (249h50m20.233487126s) [noise=true , address=167.17.72.69:4724 , piece_id=FP4XSTMRCELITDAFYCVMSCCD22BD3D4XYJGR2Z4QBQRWSXJDTD7Q, seg_pos={0 0}, upload=240175]: Done uploading piece. err: <nil> Apr 18 02:08:00.890680867 (249h51m11.484792122s) [noise=true , address=145.79.227.21:4508 , piece_id=T2OCNTKI6OZCSNTKDXLNU2IBKOC4UAYCFFIWDYRSHPYGWLQUYFZA, seg_pos={0 0}, upload=240175]: Done uploading piece. err: ecclient: upload failed (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3, address:145.79.227.21:4508): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 02:08:00.890776321 (249h51m11.484887562s) [noise=true , address=145.79.227.11:4760 , piece_id=NQRE6BQXSFUTVEOZLT75O36UQPRBCBU5BDYHGHQDXXJQ5DYWTBEA, seg_pos={0 0}, upload=240175]: Done uploading piece. err: ecclient: upload failed (node:1pgY3KuDopCrLXxvbVeNowHD9qSh2Gz7qcgUR1GpsxFkuoHhzB, address:145.79.227.11:4760): protocol: rpc: dial tcp 145.79.227.11:4760: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:08:00.891645963 (249h51m11.485757216s) [noise=true , address=145.79.227.38:4984 , piece_id=R5VAPPKIIPR7NIMTYOEHV6HNSZDWXSZRIJMF6DKLWMPKTSYQVI5A, seg_pos={0 0}, upload=240175]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:08:00.892272363 (249h51m11.48638361s) [noise=true , address=145.79.227.44:4205 , piece_id=M45SSYPWZRO6G67SFEDNSTYIOZC247SP7EUNQQYVA2TKA6DYWYJA, seg_pos={0 0}, upload=240175]: Done uploading piece. err: ecclient: upload failed (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ, address:145.79.227.44:4205): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:08:00.892471604 (249h51m11.486582847s) [seg_pos={0 0}, upload=240175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:08:00.998296578 (249h51m11.592407811s) [upload=240175]: Done uploading segments... Apr 18 02:08:00.998337329 (249h51m11.592448569s) [upload=240175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.116206706s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX..XX.X.....XX] duration: 5.00356745s trace: https://jaeger.dev.storj.io/trace/1a3372b18135033a FAIL (2026-04-18 02:07:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00359469s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.X..X.XX.XX..X.XXX..X.XXXX....X..........X..................] duration: 6.724225239s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:07:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.724321866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXX.XX] duration: 1m0.149327824s trace: https://jaeger.dev.storj.io/trace/42b13bda40357af0 FAIL (2026-04-18 02:07:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 06:00.875222314 (249h49m11.469333559s) [upload=240158]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:06:00.875228700 (249h49m11.469339944s) [upload=240158]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:06:00.984272835 (249h49m11.578384089s) [seg_pos={0 0}, upload=240158]: Begin upload segment... Apr 18 02:06:00.984490151 (249h49m11.578601404s) [seg_pos={0 0}, upload=240158]: Done begin upload segment. Apr 18 02:06:00.984585703 (249h49m11.578696956s) [noise=true , address=145.79.227.49:4349 , piece_id=WJPYL3ALQIJ2RLZWICA4VAKVNHTEVDXYE2Q6MDDHZFUFPZMT654A, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.984915334 (249h49m11.579026592s) [noise=true , address=145.79.227.38:4987 , piece_id=CPACAC467P5XMJT62FBQPKYOSGE6GPH4EKMG2DZI2VSBGMLBBCVA, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.985844999 (249h49m11.579956251s) [noise=true , address=145.79.227.11:4758 , piece_id=D7R5TAQRYAWUHMCA7KKHCGSKISGUYWSYRVRC3ANQJAMYJTOU54KA, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.985867482 (249h49m11.579978734s) [noise=true , address=145.79.227.40:4081 , piece_id=NZDYG4EVFEVJMGHLKCARM7RCXPPXGVVINRLL2DRENVXW5Y4BQV2Q, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.986452119 (249h49m11.580563368s) [noise=true , address=167.17.72.69:4743 , piece_id=Z7LC2AVAQI7EXXAVWTQ2POBB6YE4GUE3XV3RVYWXDC36RAHO3EXA, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.986625099 (249h49m11.580736352s) [noise=true , address=167.17.72.71:4794 , piece_id=NMP5XGESL4DN5YKP4KN5TCA3RUBIYX6SR632ABEMSD6V4JZRRIJA, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.986951371 (249h49m11.581062744s) [noise=true , address=5.231.73.4:4023 , piece_id=EJZRXYMX3UIO7JXZIDXO5I54GYSNIVEEMJCFEU3OZB6BP4IEGXTQ, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.987214579 (249h49m11.581325833s) [noise=true , address=79.127.205.233:4157 , piece_id=BPONCX2I7YTZAOYMWFPROYQPPMU7QYNXMCSUO6WOK2O5Z4EABN4A, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.987432437 (249h49m11.58154369s) [noise=true , address=145.79.227.15:4887 , piece_id=OGDTARYMJAMXPO65HMCCF6XX3UOWAIGJEBOUNTESYRSC4XMST7KA, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.987664642 (249h49m11.581775894s) [noise=true , address=167.17.72.70:4770 , piece_id=XTN3PS7T3I4GXEKY7GTB7MU6UWYZVDY2XIFPZIEWIYMTL5OSF7QQ, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.987912337 (249h49m11.58202359s) [noise=true , address=45.91.116.8:4063 , piece_id=ZOEJS2TTZ6BHFZUURJEKFQVUJL6WC3M5VGN3HKIVGQ4C6FLSIEIQ, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.988175795 (249h49m11.582287095s) [noise=true , address=79.127.205.237:4227 , piece_id=MTNC7R3VKGAKT7AP4OLEAUHXLIPH2SXPT3Y7QAXKWADWQ3COAGSA, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.988666600 (249h49m11.582777853s) [noise=true , address=167.17.72.68:4653 , piece_id=A4RANTYY4SEYEXY57WTNWYDKDEAQ4VMTLTDK3Y3BS6BNOGX7OR7A, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.989145396 (249h49m11.583256649s) [noise=true , address=79.127.205.235:4263 , piece_id=HGMMXUONDXPDP75NMPDBQCZVZYQR5RT7TVN7TCJLET4TX5UMFMEQ, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.989660143 (249h49m11.583771391s) [noise=true , address=145.79.227.13:4831 , piece_id=XF3IRLYB6EGWQRXXPJARX3SM2AOGD6VHZAAPXFSNZMTRPFYEMFBQ, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.990129570 (249h49m11.584240825s) [noise=true , address=193.149.164.66:4989 , piece_id=FSZBDEUI74MMIXRQLZBXVDWH6K554ROZC4EZ3ESS4EBOHFBEIMZQ, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.991227989 (249h49m11.585339239s) [noise=true , address=79.127.205.234:4194 , piece_id=J3WXXFSWXESVKG4IUO2BUVS7T5RLK6AGU7EMUHFNHCCO4LE7PMSQ, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:00.991796972 (249h49m11.585908207s) [noise=true , address=145.79.227.5:4250 , piece_id=M2YQJ2YZGZOJBFNC7UZ6N6QODNZGPW6JSEE3AVQZIXUY56TFGVYA, seg_pos={0 0}, upload=240158]: Uploading piece... Apr 18 02:06:01.079630406 (249h49m11.673741659s) [noise=true , address=145.79.227.49:4349 , piece_id=WJPYL3ALQIJ2RLZWICA4VAKVNHTEVDXYE2Q6MDDHZFUFPZMT654A, seg_pos={0 0}, upload=240158]: Done uploading piece. err: ecclient: upload failed (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk, address:145.79.227.49:4349): protocol: rpc: dial tcp 145.79.227.49:4349: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:06:01.152816240 (249h49m11.746927491s) [noise=true , address=167.17.72.68:4653 , piece_id=A4RANTYY4SEYEXY57WTNWYDKDEAQ4VMTLTDK3Y3BS6BNOGX7OR7A, seg_pos={0 0}, upload=240158]: Done uploading piece. err: ecclient: upload failed (node:12fZ7CT6Z273zoviti3xPNBPWq65PWpvoa4WXqd4VQJjFga6phX, address:167.17.72.68:4653): protocol: write tcp 139.59.144.22:53260->167.17.72.68:4653: write: connection refused; write tcp 139.59.144.22:53260->167.17.72.68:4653: write: connection refused; piecestore: piecestore close: conn closed: op Apr 18 02:06:01.594039547 (249h49m12.188150798s) [upload=240158]: Next returned nil segment Apr 18 02:06:01.594065563 (249h49m12.188176813s) [upload=240158]: Waiting for error group managing segments... Apr 18 02:06:01.744244626 (249h49m12.338355882s) [noise=true , address=145.79.227.5:4250 , piece_id=M2YQJ2YZGZOJBFNC7UZ6N6QODNZGPW6JSEE3AVQZIXUY56TFGVYA, seg_pos={0 0}, upload=240158]: Done uploading piece. err: <nil> Apr 18 02:06:01.775086192 (249h49m12.369197436s) [noise=true , address=145.79.227.40:4081 , piece_id=NZDYG4EVFEVJMGHLKCARM7RCXPPXGVVINRLL2DRENVXW5Y4BQV2Q, seg_pos={0 0}, upload=240158]: Done uploading piece. err: <nil> Apr 18 02:06:02.303606102 (249h49m12.897717352s) [noise=true , address=193.149.164.66:4989 , piece_id=FSZBDEUI74MMIXRQLZBXVDWH6K554ROZC4EZ3ESS4EBOHFBEIMZQ, seg_pos={0 0}, upload=240158]: Done uploading piece. err: <nil> Apr 18 02:06:02.396037902 (249h49m12.990149147s) [noise=true , address=79.127.205.234:4194 , piece_id=J3WXXFSWXESVKG4IUO2BUVS7T5RLK6AGU7EMUHFNHCCO4LE7PMSQ, seg_pos={0 0}, upload=240158]: Done uploading piece. err: <nil> Apr 18 02:06:02.519175970 (249h49m13.113287215s) [noise=true , address=79.127.205.235:4263 , piece_id=HGMMXUONDXPDP75NMPDBQCZVZYQR5RT7TVN7TCJLET4TX5UMFMEQ, seg_pos={0 0}, upload=240158]: Done uploading piece. err: <nil> Apr 18 02:06:02.544170306 (249h49m13.13828156s) [noise=true , address=79.127.205.237:4227 , piece_id=MTNC7R3VKGAKT7AP4OLEAUHXLIPH2SXPT3Y7QAXKWADWQ3COAGSA, seg_pos={0 0}, upload=240158]: Done uploading piece. err: <nil> Apr 18 02:06:02.770447204 (249h49m13.364558451s) [noise=true , address=79.127.205.233:4157 , piece_id=BPONCX2I7YTZAOYMWFPROYQPPMU7QYNXMCSUO6WOK2O5Z4EABN4A, seg_pos={0 0}, upload=240158]: Done uploading piece. err: <nil> Apr 18 02:06:04.297406530 (249h49m14.891517781s) [noise=true , address=5.231.73.4:4023 , piece_id=EJZRXYMX3UIO7JXZIDXO5I54GYSNIVEEMJCFEU3OZB6BP4IEGXTQ, seg_pos={0 0}, upload=240158]: Done uploading piece. err: <nil> Apr 18 02:06:05.390173157 (249h49m15.984284406s) [noise=true , address=45.91.116.8:4063 , piece_id=ZOEJS2TTZ6BHFZUURJEKFQVUJL6WC3M5VGN3HKIVGQ4C6FLSIEIQ, seg_pos={0 0}, upload=240158]: Done uploading piece. err: <nil> Apr 18 02:06:06.041667001 (249h49m16.635778269s) [noise=true , address=167.17.72.70:4770 , piece_id=XTN3PS7T3I4GXEKY7GTB7MU6UWYZVDY2XIFPZIEWIYMTL5OSF7QQ, seg_pos={0 0}, upload=240158]: Done uploading piece. err: <nil> Apr 18 02:06:07.308986818 (249h49m17.903098063s) [noise=true , address=167.17.72.69:4743 , piece_id=Z7LC2AVAQI7EXXAVWTQ2POBB6YE4GUE3XV3RVYWXDC36RAHO3EXA, seg_pos={0 0}, upload=240158]: Done uploading piece. err: <nil> Apr 18 02:06:08.069889598 (249h49m18.664000849s) [noise=true , address=145.79.227.13:4831 , piece_id=XF3IRLYB6EGWQRXXPJARX3SM2AOGD6VHZAAPXFSNZMTRPFYEMFBQ, seg_pos={0 0}, upload=240158]: Done uploading piece. err: <nil> Apr 18 02:06:09.547140625 (249h49m20.141251879s) [noise=true , address=145.79.227.15:4887 , piece_id=OGDTARYMJAMXPO65HMCCF6XX3UOWAIGJEBOUNTESYRSC4XMST7KA, seg_pos={0 0}, upload=240158]: Done uploading piece. err: <nil> Apr 18 02:07:00.625979070 (249h50m11.220090325s) [noise=true , address=145.79.227.11:4758 , piece_id=D7R5TAQRYAWUHMCA7KKHCGSKISGUYWSYRVRC3ANQJAMYJTOU54KA, seg_pos={0 0}, upload=240158]: Done uploading piece. err: ecclient: upload failed (node:1joBbXdmusUXDfe1ofLp5oXy1nVkkRAng8QGL2hAtKjDz71u5g, address:145.79.227.11:4758): protocol: rpc: dial tcp 145.79.227.11:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:07:00.626384434 (249h50m11.220495689s) [noise=true , address=145.79.227.38:4987 , piece_id=CPACAC467P5XMJT62FBQPKYOSGE6GPH4EKMG2DZI2VSBGMLBBCVA, seg_pos={0 0}, upload=240158]: Done uploading piece. err: ecclient: upload failed (node:1oZxiaBQD1grLu8d1HyERweeuXNXc1aBgyAoe1B86gqm6ypvBC, address:145.79.227.38:4987): protocol: rpc: dial tcp 145.79.227.38:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:07:00.626620292 (249h50m11.220731545s) [noise=true , address=167.17.72.71:4794 , piece_id=NMP5XGESL4DN5YKP4KN5TCA3RUBIYX6SR632ABEMSD6V4JZRRIJA, seg_pos={0 0}, upload=240158]: Done uploading piece. err: ecclient: upload failed (node:1sNGPMY8nuWVLFECNta92W6sqAkSPBKDTMALh6ozm9ro9p94cY, address:167.17.72.71:4794): protocol: write tcp 139.59.144.22:40918->167.17.72.71:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:07:00.626653983 (249h50m11.220765215s) [seg_pos={0 0}, upload=240158]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:07:00.732603698 (249h50m11.326714944s) [upload=240158]: Done uploading segments... Apr 18 02:07:00.732649714 (249h50m11.326760964s) [upload=240158]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.149348787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX..XX.X.....X] duration: 5.001796961s trace: https://jaeger.dev.storj.io/trace/381c8e88a163e6f4 FAIL (2026-04-18 02:06:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001833789s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX..XX.X.....] duration: 4.857613402s trace: https://jaeger.dev.storj.io/trace/32eb960997f77b2a FAIL (2026-04-18 02:06:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.857642145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [..X..X.XX.XX..X.XXX..X.XXXX....X..........X.................] duration: 4.88892564s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:06:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.88894377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXX.X] duration: 1m0.107835916s trace: https://jaeger.dev.storj.io/trace/4f86a607321b0459 FAIL (2026-04-18 02:06:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== .208160754s) [upload=240144]: Got next segment. Inline: false Apr 18 02:05:00.614165061 (249h48m11.208276314s) [upload=240144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:05:00.614175965 (249h48m11.208287222s) [upload=240144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:05:00.722526507 (249h48m11.316637759s) [seg_pos={0 0}, upload=240144]: Begin upload segment... Apr 18 02:05:00.722597667 (249h48m11.316708922s) [seg_pos={0 0}, upload=240144]: Done begin upload segment. Apr 18 02:05:00.722631150 (249h48m11.316742395s) [noise=true , address=79.127.205.233:4158 , piece_id=UDGL3RNYC4EAIWS63CRPTCS5SGFNXQFSNHQOKDNRUAVK65BDKLEQ, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.722673313 (249h48m11.316784563s) [noise=true , address=45.91.116.5:4318 , piece_id=U7GY6PX3FUZ5GJT4UV6AW3EZ5AB7D5TKKMTR42AFRN7OVU7QWLWQ, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.723621685 (249h48m11.317732942s) [noise=true , address=77.90.31.14:4694 , piece_id=7OMX5PKIKXWCAVYVYS2BV22QO7XQSFZBJYVSKGTGN75E3GAPO53Q, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.723863774 (249h48m11.317975035s) [noise=true , address=167.17.72.69:4725 , piece_id=YVZOZ545SN253SLNOQSHFF3B6MAFOFMBUYR66LPF2OVN5SANOPEA, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.724661984 (249h48m11.318773236s) [noise=true , address=167.17.72.68:4646 , piece_id=AGRS7AJRIQWZ4N3LNUR3OIHWIHJW75IDNNZ3YPBVGLCFA4ICQNHA, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.724922010 (249h48m11.319033263s) [noise=true , address=193.149.164.66:4995 , piece_id=4OHMCAJRULYMOG7IZKDXSRAFAFTVHEO5GKNQ5YASSE3LDUGOOW5A, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.725284669 (249h48m11.31939592s) [noise=true , address=193.149.164.71:4297 , piece_id=PW4DXIIOPUY2JMMULA7WTUW2Z3UTTOXHFELKP7ZTO2ZYSTERHFCA, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.725421459 (249h48m11.319532712s) [noise=true , address=193.149.164.143:4696 , piece_id=RRUJOLVTC2IFFNUIYHX2EEXZXPFA24ZY77IZ35UPZFXYOZTKZ2UQ, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.725705092 (249h48m11.319816343s) [noise=true , address=145.79.227.32:4840 , piece_id=COF6ULH7EXVYWA5LDONFWNIPKRUUF7W3UESU4BI6YINIL7JCLGJQ, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.725953363 (249h48m11.320064614s) [noise=true , address=167.17.72.71:4805 , piece_id=GLCEJDE2AYUOW5DX5667XWGUPGY7XNXKRFSTT4YAGGUF43IWNXCA, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.726417480 (249h48m11.320528723s) [noise=true , address=145.79.227.9:4399 , piece_id=UHKP4UVG2MKFZMZQAKMTGO3QD26YIRS7KZOROX7VLSFRYTP5VNRQ, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.726434146 (249h48m11.320545399s) [noise=true , address=79.127.205.234:4211 , piece_id=MVJEUKUVWKUDIXNLMYYSUGKKFSRZ6QW4H34K4YU4GPWNMU5AEUDQ, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.726903957 (249h48m11.321015209s) [noise=true , address=145.79.227.11:4759 , piece_id=MBOKNHNHS42NIC3Q5GYZ2PS3SYRPWX2CJRFBREPPTSNA5MXY7QOA, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.727038826 (249h48m11.321150076s) [noise=true , address=167.17.72.70:4756 , piece_id=OGSD4LCVTSNSRJN4537FSIXHFAOIKRIPW7LJRJKLUNWFJCBUSZ4Q, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.727596321 (249h48m11.321707573s) [noise=true , address=77.90.31.17:4777 , piece_id=6M2M7SHDIQXWQMRHC6X5SMR33HAF3QVIJ3E2PG3WYXFPV277SC6A, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.727671971 (249h48m11.321783221s) [noise=true , address=79.127.205.235:4253 , piece_id=XKK6YQIJQ4XI6CECLQFJG3Q56QJWL4UOZXV7LQ4RQUXVS7QG2AZQ, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.728086452 (249h48m11.322197704s) [noise=true , address=79.127.205.237:4224 , piece_id=QG3EKCUJE6F4KPEDT2UZNXIWP2WGEK3XPEFDXS23DGHSBGJFDH6Q, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.728226320 (249h48m11.322337568s) [noise=true , address=145.79.227.49:4365 , piece_id=E3UHTDVVM47ESHTRTHLLDMSULMG6VXGT577O3233I7U3COJFDDOA, seg_pos={0 0}, upload=240144]: Uploading piece... Apr 18 02:05:00.818015393 (249h48m11.41212664s) [noise=true , address=145.79.227.49:4365 , piece_id=E3UHTDVVM47ESHTRTHLLDMSULMG6VXGT577O3233I7U3COJFDDOA, seg_pos={0 0}, upload=240144]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:145.79.227.49:4365): protocol: rpc: dial tcp 145.79.227.49:4365: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:05:00.884682069 (249h48m11.478793318s) [noise=true , address=167.17.72.69:4725 , piece_id=YVZOZ545SN253SLNOQSHFF3B6MAFOFMBUYR66LPF2OVN5SANOPEA, seg_pos={0 0}, upload=240144]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:37010->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:05:00.899429861 (249h48m11.493541112s) [noise=true , address=193.149.164.143:4696 , piece_id=RRUJOLVTC2IFFNUIYHX2EEXZXPFA24ZY77IZ35UPZFXYOZTKZ2UQ, seg_pos={0 0}, upload=240144]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:38740->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:05:01.311267168 (249h48m11.905378435s) [upload=240144]: Next returned nil segment Apr 18 02:05:01.311288036 (249h48m11.90539929s) [upload=240144]: Waiting for error group managing segments... Apr 18 02:05:01.468074411 (249h48m12.062185654s) [noise=true , address=145.79.227.32:4840 , piece_id=COF6ULH7EXVYWA5LDONFWNIPKRUUF7W3UESU4BI6YINIL7JCLGJQ, seg_pos={0 0}, upload=240144]: Done uploading piece. err: <nil> Apr 18 02:05:01.469552779 (249h48m12.063664022s) [noise=true , address=45.91.116.5:4318 , piece_id=U7GY6PX3FUZ5GJT4UV6AW3EZ5AB7D5TKKMTR42AFRN7OVU7QWLWQ, seg_pos={0 0}, upload=240144]: Done uploading piece. err: <nil> Apr 18 02:05:01.871656712 (249h48m12.465767963s) [noise=true , address=167.17.72.71:4805 , piece_id=GLCEJDE2AYUOW5DX5667XWGUPGY7XNXKRFSTT4YAGGUF43IWNXCA, seg_pos={0 0}, upload=240144]: Done uploading piece. err: <nil> Apr 18 02:05:02.072220654 (249h48m12.666331905s) [noise=true , address=79.127.205.237:4224 , piece_id=QG3EKCUJE6F4KPEDT2UZNXIWP2WGEK3XPEFDXS23DGHSBGJFDH6Q, seg_pos={0 0}, upload=240144]: Done uploading piece. err: <nil> Apr 18 02:05:02.107989044 (249h48m12.702100296s) [noise=true , address=79.127.205.234:4211 , piece_id=MVJEUKUVWKUDIXNLMYYSUGKKFSRZ6QW4H34K4YU4GPWNMU5AEUDQ, seg_pos={0 0}, upload=240144]: Done uploading piece. err: <nil> Apr 18 02:05:02.108931660 (249h48m12.703042904s) [noise=true , address=79.127.205.233:4158 , piece_id=UDGL3RNYC4EAIWS63CRPTCS5SGFNXQFSNHQOKDNRUAVK65BDKLEQ, seg_pos={0 0}, upload=240144]: Done uploading piece. err: <nil> Apr 18 02:05:02.126358865 (249h48m12.720470113s) [noise=true , address=193.149.164.71:4297 , piece_id=PW4DXIIOPUY2JMMULA7WTUW2Z3UTTOXHFELKP7ZTO2ZYSTERHFCA, seg_pos={0 0}, upload=240144]: Done uploading piece. err: <nil> Apr 18 02:05:02.430143142 (249h48m13.024254387s) [noise=true , address=193.149.164.66:4995 , piece_id=4OHMCAJRULYMOG7IZKDXSRAFAFTVHEO5GKNQ5YASSE3LDUGOOW5A, seg_pos={0 0}, upload=240144]: Done uploading piece. err: <nil> Apr 18 02:05:02.465449486 (249h48m13.05956075s) [noise=true , address=79.127.205.235:4253 , piece_id=XKK6YQIJQ4XI6CECLQFJG3Q56QJWL4UOZXV7LQ4RQUXVS7QG2AZQ, seg_pos={0 0}, upload=240144]: Done uploading piece. err: <nil> Apr 18 02:05:05.802304973 (249h48m16.396416201s) [noise=true , address=77.90.31.14:4694 , piece_id=7OMX5PKIKXWCAVYVYS2BV22QO7XQSFZBJYVSKGTGN75E3GAPO53Q, seg_pos={0 0}, upload=240144]: Done uploading piece. err: <nil> Apr 18 02:05:06.231325646 (249h48m16.825436888s) [noise=true , address=77.90.31.17:4777 , piece_id=6M2M7SHDIQXWQMRHC6X5SMR33HAF3QVIJ3E2PG3WYXFPV277SC6A, seg_pos={0 0}, upload=240144]: Done uploading piece. err: <nil> Apr 18 02:05:09.280155297 (249h48m19.874266536s) [noise=true , address=145.79.227.9:4399 , piece_id=UHKP4UVG2MKFZMZQAKMTGO3QD26YIRS7KZOROX7VLSFRYTP5VNRQ, seg_pos={0 0}, upload=240144]: Done uploading piece. err: <nil> Apr 18 02:05:12.837122778 (249h48m23.431234021s) [noise=true , address=167.17.72.70:4756 , piece_id=OGSD4LCVTSNSRJN4537FSIXHFAOIKRIPW7LJRJKLUNWFJCBUSZ4Q, seg_pos={0 0}, upload=240144]: Done uploading piece. err: <nil> Apr 18 02:06:00.330069318 (249h49m10.924180564s) [noise=true , address=145.79.227.11:4759 , piece_id=MBOKNHNHS42NIC3Q5GYZ2PS3SYRPWX2CJRFBREPPTSNA5MXY7QOA, seg_pos={0 0}, upload=240144]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:145.79.227.11:4759): protocol: rpc: dial tcp 145.79.227.11:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:06:00.330300967 (249h49m10.924412221s) [noise=true , address=167.17.72.68:4646 , piece_id=AGRS7AJRIQWZ4N3LNUR3OIHWIHJW75IDNNZ3YPBVGLCFA4ICQNHA, seg_pos={0 0}, upload=240144]: Done uploading piece. err: ecclient: upload failed (node:12PqAjcGb2g9w5yHp6Sd432wofz9jKWK9hRRTxLzzv2BAFL1iRW, address:167.17.72.68:4646): protocol: write tcp 139.59.144.22:46626->167.17.72.68:4646: use of closed network connection; conn closed: op Apr 18 02:06:00.330362333 (249h49m10.924473562s) [seg_pos={0 0}, upload=240144]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:06:00.434981360 (249h49m11.029092606s) [upload=240144]: Done uploading segments... Apr 18 02:06:00.435024729 (249h49m11.029135978s) [upload=240144]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.107868584s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX..XX.X....] duration: 3.227112868s trace: https://jaeger.dev.storj.io/trace/32eb960997f77b2a FAIL (2026-04-18 02:05:43 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.227158941s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [X..X..X.XX.XX..X.XXX..X.XXXX....X..........X................] duration: 7.154268503s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:05:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.154310896s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX..XX.X...] duration: 4.103532958s trace: https://jaeger.dev.storj.io/trace/32eb960997f77b2a FAIL (2026-04-18 02:04:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.103567737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [XX..X..X.XX.XX..X.XXX..X.XXXX....X..........X...............] duration: 10.715259749s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:04:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.715296073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXX.] duration: 22.313317602s trace: https://jaeger.dev.storj.io/trace/771331a24c29be8f FAIL (2026-04-18 02:04:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 786 (249h46m2.413083032s) [upload=240108]: Done starting upload Apr 18 02:02:51.819205799 (249h46m2.413317032s) [upload=240108]: Uploading segments... Apr 18 02:02:51.819293621 (249h46m2.41340486s) [upload=240108]: Got next segment. Inline: false Apr 18 02:02:51.819360428 (249h46m2.413471666s) [upload=240108]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:02:51.819367447 (249h46m2.413478686s) [upload=240108]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:02:51.930405180 (249h46m2.524516419s) [seg_pos={0 0}, upload=240108]: Begin upload segment... Apr 18 02:02:51.930464271 (249h46m2.524575511s) [seg_pos={0 0}, upload=240108]: Done begin upload segment. Apr 18 02:02:51.930535315 (249h46m2.52464656s) [noise=true , address=167.17.72.70:4757 , piece_id=GZUBTURHPDUNQMMMRNZIVUW6UXQEFWZFCTVEDS3SQAGCTXYONU7A, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.930566475 (249h46m2.524677718s) [noise=true , address=145.79.227.11:4760 , piece_id=ZI6PLQXUQHUOMUM3JHPGB2JDAAYOBZELGNPN4GGJ7HZGL74FVS6A, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.931184081 (249h46m2.525295321s) [noise=true , address=193.149.164.71:4316 , piece_id=4AQUIVXSTC4G4FCQ2X5NJRZ7SJVVXCCCZPWENBCJWS3S7ZPS6IYA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.931267272 (249h46m2.52537851s) [noise=true , address=145.79.227.43:4184 , piece_id=6DSB54N5MROQRPML76TFMJ6XWSJSWCWYD5H7C72BPBQJIYWLKJBA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.932526078 (249h46m2.526637323s) [noise=true , address=193.149.164.66:4969 , piece_id=TBE4DSNEJ2WLQSKEEIYWFYKEB6YL3RKTIWMACQPMDHBCLHTBZMEQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.932722901 (249h46m2.52683414s) [noise=true , address=145.79.227.51:4424 , piece_id=ZLH2XZOFDZSRPEXFV2R6WQI536MSC35V6NST2ZLS5UO6MENOZM6Q, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.933518263 (249h46m2.527629503s) [noise=true , address=45.91.116.8:4063 , piece_id=ZHHGUXU6LZ4L7V55LGF6CN2JRX5AWRKAGIGVM7FXGSYZJMUPQWKA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.933633359 (249h46m2.527744598s) [noise=true , address=79.127.205.233:4169 , piece_id=FLY4XEASUNAJ6FJ7OAY3653QKZ2CMZ3ZZ7V5U7BKX7DJK4UIKBFQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.934508268 (249h46m2.528619511s) [noise=true , address=145.79.227.38:4004 , piece_id=6RXHBN547GQUHSFJCTN53N6J6H4AWSO6XN3XEAIUFQW3JJ4GHAIA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.934983919 (249h46m2.529095169s) [noise=true , address=79.127.205.234:4212 , piece_id=ISB6K35X3WZFIPNLYWZTXPLY5R2H7KFIRARLYS3F7TG7CZB3QFEA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.935154464 (249h46m2.529265703s) [noise=true , address=167.17.72.69:4728 , piece_id=YDPH3DETIYSUKVZIUMKQF34T3F2CWPAVBYXMZYY7VISRH5VOX4PQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.935451065 (249h46m2.529562318s) [noise=true , address=145.79.227.18:4031 , piece_id=THLGVEBSKHN4Y22ZU6R3KCZRXSCE75SSHXV2ERMHMWZFYDKJW27Q, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.935605097 (249h46m2.529716339s) [noise=true , address=79.127.205.237:4242 , piece_id=AVRMDRHAGKZQ3XETCHSG7NVQZMAVQJQWSO7L6HPY6RTEI72YM5SQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.935893844 (249h46m2.530005087s) [noise=true , address=145.79.227.13:4817 , piece_id=SXIO3IMUZEVGPEYQOGHN3V7WEO2SVDTQNJBCF3DTXNPURZR4RUTA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.936098039 (249h46m2.53020929s) [noise=true , address=77.90.31.17:4794 , piece_id=676CYSXKGTQBHEQRE35TFOWTQPYFEUEPDE34MVEWMY7VWO3DVSXA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.936777644 (249h46m2.530888894s) [noise=true , address=5.231.73.4:4023 , piece_id=LI5AAZZBIMEUDKEZ6KQMR5BUIAGS4LOKD5AV3VE6AARJZEF2L6HQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.937369225 (249h46m2.531480472s) [noise=true , address=145.79.227.34:4912 , piece_id=VF5MFFM5HKQK75A7Q2MRBPHWFWIJ65L6KBSPNJGELK4B5NUXFKAQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.938710118 (249h46m2.53282136s) [noise=true , address=145.79.227.15:4903 , piece_id=FW4DRA4YYH5XEQ5IWIMIGPMLPBZZRO2CHYVU7KGJS2MHM7QIIALQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:52.520512470 (249h46m3.11462372s) [upload=240108]: Next returned nil segment Apr 18 02:02:52.520535481 (249h46m3.114646738s) [upload=240108]: Waiting for error group managing segments... Apr 18 02:02:52.667224947 (249h46m3.261336194s) [noise=true , address=145.79.227.18:4031 , piece_id=THLGVEBSKHN4Y22ZU6R3KCZRXSCE75SSHXV2ERMHMWZFYDKJW27Q, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:52.681310784 (249h46m3.275422043s) [noise=true , address=145.79.227.43:4184 , piece_id=6DSB54N5MROQRPML76TFMJ6XWSJSWCWYD5H7C72BPBQJIYWLKJBA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:52.728187945 (249h46m3.322299188s) [noise=true , address=145.79.227.15:4903 , piece_id=FW4DRA4YYH5XEQ5IWIMIGPMLPBZZRO2CHYVU7KGJS2MHM7QIIALQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:52.974149290 (249h46m3.568260554s) [noise=true , address=45.91.116.8:4063 , piece_id=ZHHGUXU6LZ4L7V55LGF6CN2JRX5AWRKAGIGVM7FXGSYZJMUPQWKA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: storage node overloaded, request limit: 2000 Apr 18 02:02:53.193164936 (249h46m3.787276179s) [noise=true , address=193.149.164.71:4316 , piece_id=4AQUIVXSTC4G4FCQ2X5NJRZ7SJVVXCCCZPWENBCJWS3S7ZPS6IYA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:53.301307623 (249h46m3.895418872s) [noise=true , address=79.127.205.234:4212 , piece_id=ISB6K35X3WZFIPNLYWZTXPLY5R2H7KFIRARLYS3F7TG7CZB3QFEA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:53.334528383 (249h46m3.928639627s) [noise=true , address=5.231.73.4:4023 , piece_id=LI5AAZZBIMEUDKEZ6KQMR5BUIAGS4LOKD5AV3VE6AARJZEF2L6HQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:53.399804501 (249h46m3.993915748s) [noise=true , address=79.127.205.237:4242 , piece_id=AVRMDRHAGKZQ3XETCHSG7NVQZMAVQJQWSO7L6HPY6RTEI72YM5SQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:53.430371883 (249h46m4.024483122s) [noise=true , address=193.149.164.66:4969 , piece_id=TBE4DSNEJ2WLQSKEEIYWFYKEB6YL3RKTIWMACQPMDHBCLHTBZMEQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:53.652237316 (249h46m4.246348564s) [noise=true , address=79.127.205.233:4169 , piece_id=FLY4XEASUNAJ6FJ7OAY3653QKZ2CMZ3ZZ7V5U7BKX7DJK4UIKBFQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:55.187290060 (249h46m5.781401308s) [noise=true , address=145.79.227.13:4817 , piece_id=SXIO3IMUZEVGPEYQOGHN3V7WEO2SVDTQNJBCF3DTXNPURZR4RUTA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:56.970873983 (249h46m7.564985227s) [noise=true , address=167.17.72.69:4728 , piece_id=YDPH3DETIYSUKVZIUMKQF34T3F2CWPAVBYXMZYY7VISRH5VOX4PQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:57.821678829 (249h46m8.415790068s) [noise=true , address=145.79.227.51:4424 , piece_id=ZLH2XZOFDZSRPEXFV2R6WQI536MSC35V6NST2ZLS5UO6MENOZM6Q, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:03:03.873286729 (249h46m14.467397989s) [noise=true , address=77.90.31.17:4794 , piece_id=676CYSXKGTQBHEQRE35TFOWTQPYFEUEPDE34MVEWMY7VWO3DVSXA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:03:51.528031596 (249h47m2.122142844s) [noise=true , address=167.17.72.70:4757 , piece_id=GZUBTURHPDUNQMMMRNZIVUW6UXQEFWZFCTVEDS3SQAGCTXYONU7A, seg_pos={0 0}, upload=240108]: Done uploading piece. err: ecclient: upload failed (node:12d8BNwjyGM2XzHNiijZM17NcKmPfDKUXrmuxmuD5n7wcNXHDBk, address:167.17.72.70:4757): protocol: rpc: dial tcp 167.17.72.70:4757: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:03:51.528058696 (249h47m2.122169945s) [noise=true , address=145.79.227.11:4760 , piece_id=ZI6PLQXUQHUOMUM3JHPGB2JDAAYOBZELGNPN4GGJ7HZGL74FVS6A, seg_pos={0 0}, upload=240108]: Done uploading piece. err: ecclient: upload failed (node:1pgY3KuDopCrLXxvbVeNowHD9qSh2Gz7qcgUR1GpsxFkuoHhzB, address:145.79.227.11:4760): protocol: rpc: dial tcp 145.79.227.11:4760: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:03:51.528430620 (249h47m2.122541869s) [noise=true , address=145.79.227.34:4912 , piece_id=VF5MFFM5HKQK75A7Q2MRBPHWFWIJ65L6KBSPNJGELK4B5NUXFKAQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: rpc: dial tcp 145.79.227.34:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:03:51.528713048 (249h47m2.122824299s) [noise=true , address=145.79.227.38:4004 , piece_id=6RXHBN547GQUHSFJCTN53N6J6H4AWSO6XN3XEAIUFQW3JJ4GHAIA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: ecclient: upload failed (node:1iSAPoDgvJmcbXpHKpVkpNHzso3rvFX5PJa8PooBmohS3BgGKZ, address:145.79.227.38:4004): protocol: rpc: dial tcp 145.79.227.38:4004: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:03:51.528749895 (249h47m2.122861157s) [seg_pos={0 0}, upload=240108]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:03:51.660624478 (249h47m2.254735721s) [upload=240108]: Done uploading segments... Apr 18 02:03:51.660670936 (249h47m2.254782176s) [upload=240108]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.31334947s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX..XX.X..] duration: 3.919507393s trace: https://jaeger.dev.storj.io/trace/32eb960997f77b2a FAIL (2026-04-18 02:03:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.919532255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXX] duration: 1m0.134617958s trace: https://jaeger.dev.storj.io/trace/771331a24c29be8f FAIL (2026-04-18 02:03:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 786 (249h46m2.413083032s) [upload=240108]: Done starting upload Apr 18 02:02:51.819205799 (249h46m2.413317032s) [upload=240108]: Uploading segments... Apr 18 02:02:51.819293621 (249h46m2.41340486s) [upload=240108]: Got next segment. Inline: false Apr 18 02:02:51.819360428 (249h46m2.413471666s) [upload=240108]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:02:51.819367447 (249h46m2.413478686s) [upload=240108]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:02:51.930405180 (249h46m2.524516419s) [seg_pos={0 0}, upload=240108]: Begin upload segment... Apr 18 02:02:51.930464271 (249h46m2.524575511s) [seg_pos={0 0}, upload=240108]: Done begin upload segment. Apr 18 02:02:51.930535315 (249h46m2.52464656s) [noise=true , address=167.17.72.70:4757 , piece_id=GZUBTURHPDUNQMMMRNZIVUW6UXQEFWZFCTVEDS3SQAGCTXYONU7A, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.930566475 (249h46m2.524677718s) [noise=true , address=145.79.227.11:4760 , piece_id=ZI6PLQXUQHUOMUM3JHPGB2JDAAYOBZELGNPN4GGJ7HZGL74FVS6A, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.931184081 (249h46m2.525295321s) [noise=true , address=193.149.164.71:4316 , piece_id=4AQUIVXSTC4G4FCQ2X5NJRZ7SJVVXCCCZPWENBCJWS3S7ZPS6IYA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.931267272 (249h46m2.52537851s) [noise=true , address=145.79.227.43:4184 , piece_id=6DSB54N5MROQRPML76TFMJ6XWSJSWCWYD5H7C72BPBQJIYWLKJBA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.932526078 (249h46m2.526637323s) [noise=true , address=193.149.164.66:4969 , piece_id=TBE4DSNEJ2WLQSKEEIYWFYKEB6YL3RKTIWMACQPMDHBCLHTBZMEQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.932722901 (249h46m2.52683414s) [noise=true , address=145.79.227.51:4424 , piece_id=ZLH2XZOFDZSRPEXFV2R6WQI536MSC35V6NST2ZLS5UO6MENOZM6Q, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.933518263 (249h46m2.527629503s) [noise=true , address=45.91.116.8:4063 , piece_id=ZHHGUXU6LZ4L7V55LGF6CN2JRX5AWRKAGIGVM7FXGSYZJMUPQWKA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.933633359 (249h46m2.527744598s) [noise=true , address=79.127.205.233:4169 , piece_id=FLY4XEASUNAJ6FJ7OAY3653QKZ2CMZ3ZZ7V5U7BKX7DJK4UIKBFQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.934508268 (249h46m2.528619511s) [noise=true , address=145.79.227.38:4004 , piece_id=6RXHBN547GQUHSFJCTN53N6J6H4AWSO6XN3XEAIUFQW3JJ4GHAIA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.934983919 (249h46m2.529095169s) [noise=true , address=79.127.205.234:4212 , piece_id=ISB6K35X3WZFIPNLYWZTXPLY5R2H7KFIRARLYS3F7TG7CZB3QFEA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.935154464 (249h46m2.529265703s) [noise=true , address=167.17.72.69:4728 , piece_id=YDPH3DETIYSUKVZIUMKQF34T3F2CWPAVBYXMZYY7VISRH5VOX4PQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.935451065 (249h46m2.529562318s) [noise=true , address=145.79.227.18:4031 , piece_id=THLGVEBSKHN4Y22ZU6R3KCZRXSCE75SSHXV2ERMHMWZFYDKJW27Q, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.935605097 (249h46m2.529716339s) [noise=true , address=79.127.205.237:4242 , piece_id=AVRMDRHAGKZQ3XETCHSG7NVQZMAVQJQWSO7L6HPY6RTEI72YM5SQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.935893844 (249h46m2.530005087s) [noise=true , address=145.79.227.13:4817 , piece_id=SXIO3IMUZEVGPEYQOGHN3V7WEO2SVDTQNJBCF3DTXNPURZR4RUTA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.936098039 (249h46m2.53020929s) [noise=true , address=77.90.31.17:4794 , piece_id=676CYSXKGTQBHEQRE35TFOWTQPYFEUEPDE34MVEWMY7VWO3DVSXA, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.936777644 (249h46m2.530888894s) [noise=true , address=5.231.73.4:4023 , piece_id=LI5AAZZBIMEUDKEZ6KQMR5BUIAGS4LOKD5AV3VE6AARJZEF2L6HQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.937369225 (249h46m2.531480472s) [noise=true , address=145.79.227.34:4912 , piece_id=VF5MFFM5HKQK75A7Q2MRBPHWFWIJ65L6KBSPNJGELK4B5NUXFKAQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:51.938710118 (249h46m2.53282136s) [noise=true , address=145.79.227.15:4903 , piece_id=FW4DRA4YYH5XEQ5IWIMIGPMLPBZZRO2CHYVU7KGJS2MHM7QIIALQ, seg_pos={0 0}, upload=240108]: Uploading piece... Apr 18 02:02:52.520512470 (249h46m3.11462372s) [upload=240108]: Next returned nil segment Apr 18 02:02:52.520535481 (249h46m3.114646738s) [upload=240108]: Waiting for error group managing segments... Apr 18 02:02:52.667224947 (249h46m3.261336194s) [noise=true , address=145.79.227.18:4031 , piece_id=THLGVEBSKHN4Y22ZU6R3KCZRXSCE75SSHXV2ERMHMWZFYDKJW27Q, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:52.681310784 (249h46m3.275422043s) [noise=true , address=145.79.227.43:4184 , piece_id=6DSB54N5MROQRPML76TFMJ6XWSJSWCWYD5H7C72BPBQJIYWLKJBA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:52.728187945 (249h46m3.322299188s) [noise=true , address=145.79.227.15:4903 , piece_id=FW4DRA4YYH5XEQ5IWIMIGPMLPBZZRO2CHYVU7KGJS2MHM7QIIALQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:52.974149290 (249h46m3.568260554s) [noise=true , address=45.91.116.8:4063 , piece_id=ZHHGUXU6LZ4L7V55LGF6CN2JRX5AWRKAGIGVM7FXGSYZJMUPQWKA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: storage node overloaded, request limit: 2000 Apr 18 02:02:53.193164936 (249h46m3.787276179s) [noise=true , address=193.149.164.71:4316 , piece_id=4AQUIVXSTC4G4FCQ2X5NJRZ7SJVVXCCCZPWENBCJWS3S7ZPS6IYA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:53.301307623 (249h46m3.895418872s) [noise=true , address=79.127.205.234:4212 , piece_id=ISB6K35X3WZFIPNLYWZTXPLY5R2H7KFIRARLYS3F7TG7CZB3QFEA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:53.334528383 (249h46m3.928639627s) [noise=true , address=5.231.73.4:4023 , piece_id=LI5AAZZBIMEUDKEZ6KQMR5BUIAGS4LOKD5AV3VE6AARJZEF2L6HQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:53.399804501 (249h46m3.993915748s) [noise=true , address=79.127.205.237:4242 , piece_id=AVRMDRHAGKZQ3XETCHSG7NVQZMAVQJQWSO7L6HPY6RTEI72YM5SQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:53.430371883 (249h46m4.024483122s) [noise=true , address=193.149.164.66:4969 , piece_id=TBE4DSNEJ2WLQSKEEIYWFYKEB6YL3RKTIWMACQPMDHBCLHTBZMEQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:53.652237316 (249h46m4.246348564s) [noise=true , address=79.127.205.233:4169 , piece_id=FLY4XEASUNAJ6FJ7OAY3653QKZ2CMZ3ZZ7V5U7BKX7DJK4UIKBFQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:55.187290060 (249h46m5.781401308s) [noise=true , address=145.79.227.13:4817 , piece_id=SXIO3IMUZEVGPEYQOGHN3V7WEO2SVDTQNJBCF3DTXNPURZR4RUTA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:56.970873983 (249h46m7.564985227s) [noise=true , address=167.17.72.69:4728 , piece_id=YDPH3DETIYSUKVZIUMKQF34T3F2CWPAVBYXMZYY7VISRH5VOX4PQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:02:57.821678829 (249h46m8.415790068s) [noise=true , address=145.79.227.51:4424 , piece_id=ZLH2XZOFDZSRPEXFV2R6WQI536MSC35V6NST2ZLS5UO6MENOZM6Q, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:03:03.873286729 (249h46m14.467397989s) [noise=true , address=77.90.31.17:4794 , piece_id=676CYSXKGTQBHEQRE35TFOWTQPYFEUEPDE34MVEWMY7VWO3DVSXA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: <nil> Apr 18 02:03:51.528031596 (249h47m2.122142844s) [noise=true , address=167.17.72.70:4757 , piece_id=GZUBTURHPDUNQMMMRNZIVUW6UXQEFWZFCTVEDS3SQAGCTXYONU7A, seg_pos={0 0}, upload=240108]: Done uploading piece. err: ecclient: upload failed (node:12d8BNwjyGM2XzHNiijZM17NcKmPfDKUXrmuxmuD5n7wcNXHDBk, address:167.17.72.70:4757): protocol: rpc: dial tcp 167.17.72.70:4757: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:03:51.528058696 (249h47m2.122169945s) [noise=true , address=145.79.227.11:4760 , piece_id=ZI6PLQXUQHUOMUM3JHPGB2JDAAYOBZELGNPN4GGJ7HZGL74FVS6A, seg_pos={0 0}, upload=240108]: Done uploading piece. err: ecclient: upload failed (node:1pgY3KuDopCrLXxvbVeNowHD9qSh2Gz7qcgUR1GpsxFkuoHhzB, address:145.79.227.11:4760): protocol: rpc: dial tcp 145.79.227.11:4760: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:03:51.528430620 (249h47m2.122541869s) [noise=true , address=145.79.227.34:4912 , piece_id=VF5MFFM5HKQK75A7Q2MRBPHWFWIJ65L6KBSPNJGELK4B5NUXFKAQ, seg_pos={0 0}, upload=240108]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: rpc: dial tcp 145.79.227.34:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:03:51.528713048 (249h47m2.122824299s) [noise=true , address=145.79.227.38:4004 , piece_id=6RXHBN547GQUHSFJCTN53N6J6H4AWSO6XN3XEAIUFQW3JJ4GHAIA, seg_pos={0 0}, upload=240108]: Done uploading piece. err: ecclient: upload failed (node:1iSAPoDgvJmcbXpHKpVkpNHzso3rvFX5PJa8PooBmohS3BgGKZ, address:145.79.227.38:4004): protocol: rpc: dial tcp 145.79.227.38:4004: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:03:51.528749895 (249h47m2.122861157s) [seg_pos={0 0}, upload=240108]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:03:51.660624478 (249h47m2.254735721s) [upload=240108]: Done uploading segments... Apr 18 02:03:51.660670936 (249h47m2.254782176s) [upload=240108]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.134633759s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X..............] duration: 8.201054364s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:03:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.201085756s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X.............] duration: 5.298357241s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:03:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.298390873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX..XX.X.] duration: 4.873315931s trace: https://jaeger.dev.storj.io/trace/32eb960997f77b2a FAIL (2026-04-18 02:03:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.87334316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XX] duration: 1m0.10592619s trace: https://jaeger.dev.storj.io/trace/7f5b068f326097b8 FAIL (2026-04-18 02:02:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 6446 (249h45m2.253127737s) [seg_pos={0 0}, upload=240090]: Done begin upload segment. Apr 18 02:01:51.659077610 (249h45m2.253188851s) [noise=true , address=145.79.227.32:4859 , piece_id=6BAOZMN2VGCES6Y5KYQRJPOKMRZ6XE5KZ6HTVR6DYI4DYXHLF3XQ, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.659974095 (249h45m2.254085339s) [noise=true , address=167.17.72.70:4751 , piece_id=PE7MUREOEQQV4PHJ3HISYTSKN6TQWWN4FA5BSBETC6GDPXQU5VXQ, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.660024549 (249h45m2.254135798s) [noise=true , address=193.149.164.66:4996 , piece_id=PWROLP3N45RUNFVC36F6X2ITEUBUF7BAXR4S3YUGE2NLNGCZJPUA, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.660844273 (249h45m2.254955521s) [noise=true , address=77.90.31.17:4797 , piece_id=XASUURKTLB3PCYKE3MPWJBSNPWHNYAMMQPHVJPKJQOTED5HLQO5Q, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.660896068 (249h45m2.255007319s) [noise=true , address=145.79.227.34:4938 , piece_id=KRE3BAY6OKVUFM5OXMI2MF3P4OZCEPKFYHADI3VCC5MJPN2QTXGQ, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.661554134 (249h45m2.255665384s) [noise=true , address=79.127.205.237:4239 , piece_id=2TUKT63DCOSQQ3UCZKRGMM4EDOHKH7BDEYDKRA65RKTEX7IYHDWA, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.661634406 (249h45m2.255745655s) [noise=true , address=193.149.164.71:4314 , piece_id=4CVGED336OXWQY5YLVOOVQSHYGYQOTUAW4PL2G6JZYZS6BPKJOFQ, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.662282536 (249h45m2.256393782s) [noise=true , address=193.149.164.143:4696 , piece_id=PNPA4ESW4NHH4QSU5UOVF75O3S3GNTY27ZLAM3JCFDKQNTDTOUSA, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.662443945 (249h45m2.25655519s) [noise=true , address=167.17.72.69:4711 , piece_id=2ULBMKDIQA2HGGXK7MKJUOVSLSEESRWDM7BPDJ4JQVWLYG76OEDA, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.662987922 (249h45m2.257099164s) [noise=true , address=167.17.72.71:4808 , piece_id=JCNKBIPK5QKK2WSRM6IPPVQBDXBCY3BG44YFKGXGZ4WET7OAGXGA, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.663104284 (249h45m2.257215535s) [noise=true , address=167.17.72.68:4648 , piece_id=7QTCKEQEJ3WZWV7Y7LLJB2KZLODQH65EJXB6TN237OOG5N445GYA, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.663698254 (249h45m2.257809505s) [noise=true , address=79.127.205.233:4171 , piece_id=YV2UZ4Q4GWO2CQR7PCZ3DXY3E46MPCGYILVI5XV5A7MXC34NNFWQ, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.663796111 (249h45m2.257907354s) [noise=true , address=79.127.205.234:4181 , piece_id=FOOT7F2JKFTIEEH22Z7QOJKRTMFAUVGWPEYPICEVWBXV44DBBWFA, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.664267815 (249h45m2.258379059s) [noise=true , address=5.231.73.5:4007 , piece_id=46JUPVMIUDUJKI5KS5EMO6M4L2RNLGKVO5S3TTLER6UPCDEEFJEQ, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.664473133 (249h45m2.258584374s) [noise=true , address=145.79.227.11:4758 , piece_id=UAEL5AJDOO4Q2CYUWLNNNRZ53ZOBAXYFTKC36NKLS2PODJN44BDQ, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.665020915 (249h45m2.259132163s) [noise=true , address=145.79.227.43:4179 , piece_id=BPZZ43MALUQTOXBRZQBUFZDKB6MWZ454EGORIQWXB7DK2OY67W4Q, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.666262864 (249h45m2.260374108s) [noise=true , address=145.79.227.33:4900 , piece_id=G26ALVYRYHSG6O6YFWNGBD4RMDEVS42JG4SNE62U5UTWPSDY2DEQ, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.666898484 (249h45m2.261009725s) [noise=true , address=79.127.205.235:4280 , piece_id=BZFFDJNAG7H7DCICKDGMS3LAKR56733UYPFGCAGMRDLEEF5OC23A, seg_pos={0 0}, upload=240090]: Uploading piece... Apr 18 02:01:51.809189914 (249h45m2.403301165s) [noise=true , address=167.17.72.70:4751 , piece_id=PE7MUREOEQQV4PHJ3HISYTSKN6TQWWN4FA5BSBETC6GDPXQU5VXQ, seg_pos={0 0}, upload=240090]: Done uploading piece. err: ecclient: upload failed (node:1yUJd7aXjqiB9ykfCkt3wMj2Pn3qKNbjEwgtLk4KqYfFyaecep, address:167.17.72.70:4751): protocol: write tcp 139.59.144.22:41706->167.17.72.70:4751: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:01:51.814409996 (249h45m2.40852124s) [noise=true , address=167.17.72.69:4711 , piece_id=2ULBMKDIQA2HGGXK7MKJUOVSLSEESRWDM7BPDJ4JQVWLYG76OEDA, seg_pos={0 0}, upload=240090]: Done uploading piece. err: ecclient: upload failed (node:12pefcoMB9X8CRc3X8CgJMF9F3uyZkQKDFHirCBM56PiWzLE9hZ, address:167.17.72.69:4711): protocol: write tcp 139.59.144.22:60466->167.17.72.69:4711: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:01:51.816768810 (249h45m2.410880062s) [noise=true , address=167.17.72.71:4808 , piece_id=JCNKBIPK5QKK2WSRM6IPPVQBDXBCY3BG44YFKGXGZ4WET7OAGXGA, seg_pos={0 0}, upload=240090]: Done uploading piece. err: ecclient: upload failed (node:12M2MMAvXA6ds3zQSMpwqqc5RXDthj8KgbopnV2adDrks7iKRL5, address:167.17.72.71:4808): protocol: write tcp 139.59.144.22:53818->167.17.72.71:4808: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:01:51.824105228 (249h45m2.418216477s) [noise=true , address=193.149.164.143:4696 , piece_id=PNPA4ESW4NHH4QSU5UOVF75O3S3GNTY27ZLAM3JCFDKQNTDTOUSA, seg_pos={0 0}, upload=240090]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:53116->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:01:52.230463700 (249h45m2.824574943s) [upload=240090]: Next returned nil segment Apr 18 02:01:52.230480875 (249h45m2.824592126s) [upload=240090]: Waiting for error group managing segments... Apr 18 02:01:52.417124406 (249h45m3.011235648s) [noise=true , address=145.79.227.43:4179 , piece_id=BPZZ43MALUQTOXBRZQBUFZDKB6MWZ454EGORIQWXB7DK2OY67W4Q, seg_pos={0 0}, upload=240090]: Done uploading piece. err: <nil> Apr 18 02:01:52.538039487 (249h45m3.132150736s) [noise=true , address=5.231.73.5:4007 , piece_id=46JUPVMIUDUJKI5KS5EMO6M4L2RNLGKVO5S3TTLER6UPCDEEFJEQ, seg_pos={0 0}, upload=240090]: Done uploading piece. err: <nil> Apr 18 02:01:52.950301089 (249h45m3.54441234s) [noise=true , address=193.149.164.71:4314 , piece_id=4CVGED336OXWQY5YLVOOVQSHYGYQOTUAW4PL2G6JZYZS6BPKJOFQ, seg_pos={0 0}, upload=240090]: Done uploading piece. err: <nil> Apr 18 02:01:53.054722905 (249h45m3.648834149s) [noise=true , address=79.127.205.234:4181 , piece_id=FOOT7F2JKFTIEEH22Z7QOJKRTMFAUVGWPEYPICEVWBXV44DBBWFA, seg_pos={0 0}, upload=240090]: Done uploading piece. err: <nil> Apr 18 02:01:53.075915781 (249h45m3.670027034s) [noise=true , address=79.127.205.237:4239 , piece_id=2TUKT63DCOSQQ3UCZKRGMM4EDOHKH7BDEYDKRA65RKTEX7IYHDWA, seg_pos={0 0}, upload=240090]: Done uploading piece. err: <nil> Apr 18 02:01:53.113476186 (249h45m3.70758743s) [noise=true , address=79.127.205.233:4171 , piece_id=YV2UZ4Q4GWO2CQR7PCZ3DXY3E46MPCGYILVI5XV5A7MXC34NNFWQ, seg_pos={0 0}, upload=240090]: Done uploading piece. err: <nil> Apr 18 02:01:53.148992461 (249h45m3.743103705s) [noise=true , address=193.149.164.66:4996 , piece_id=PWROLP3N45RUNFVC36F6X2ITEUBUF7BAXR4S3YUGE2NLNGCZJPUA, seg_pos={0 0}, upload=240090]: Done uploading piece. err: <nil> Apr 18 02:01:53.171362483 (249h45m3.765473713s) [noise=true , address=79.127.205.235:4280 , piece_id=BZFFDJNAG7H7DCICKDGMS3LAKR56733UYPFGCAGMRDLEEF5OC23A, seg_pos={0 0}, upload=240090]: Done uploading piece. err: <nil> Apr 18 02:01:55.841553029 (249h45m6.435664276s) [noise=true , address=145.79.227.32:4859 , piece_id=6BAOZMN2VGCES6Y5KYQRJPOKMRZ6XE5KZ6HTVR6DYI4DYXHLF3XQ, seg_pos={0 0}, upload=240090]: Done uploading piece. err: <nil> Apr 18 02:01:56.939643416 (249h45m7.533754655s) [noise=true , address=167.17.72.68:4648 , piece_id=7QTCKEQEJ3WZWV7Y7LLJB2KZLODQH65EJXB6TN237OOG5N445GYA, seg_pos={0 0}, upload=240090]: Done uploading piece. err: <nil> Apr 18 02:01:58.310014118 (249h45m8.904125357s) [noise=true , address=77.90.31.17:4797 , piece_id=XASUURKTLB3PCYKE3MPWJBSNPWHNYAMMQPHVJPKJQOTED5HLQO5Q, seg_pos={0 0}, upload=240090]: Done uploading piece. err: <nil> Apr 18 02:02:51.266584157 (249h46m1.860695406s) [noise=true , address=145.79.227.33:4900 , piece_id=G26ALVYRYHSG6O6YFWNGBD4RMDEVS42JG4SNE62U5UTWPSDY2DEQ, seg_pos={0 0}, upload=240090]: Done uploading piece. err: ecclient: upload failed (node:1R8K65zsCspu7W2oAnvtpz7zEsQFkDG1e5BAp6faQYMQrkh4oQ, address:145.79.227.33:4900): protocol: rpc: dial tcp 145.79.227.33:4900: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:02:51.266862598 (249h46m1.860973848s) [noise=true , address=145.79.227.11:4758 , piece_id=UAEL5AJDOO4Q2CYUWLNNNRZ53ZOBAXYFTKC36NKLS2PODJN44BDQ, seg_pos={0 0}, upload=240090]: Done uploading piece. err: ecclient: upload failed (node:1joBbXdmusUXDfe1ofLp5oXy1nVkkRAng8QGL2hAtKjDz71u5g, address:145.79.227.11:4758): protocol: rpc: dial tcp 145.79.227.11:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:02:51.267068326 (249h46m1.861179573s) [noise=true , address=145.79.227.34:4938 , piece_id=KRE3BAY6OKVUFM5OXMI2MF3P4OZCEPKFYHADI3VCC5MJPN2QTXGQ, seg_pos={0 0}, upload=240090]: Done uploading piece. err: ecclient: upload failed (node:128NJghaGo1dYxjSyjWksexXS89mgdtPH4GaTrwSavrqNfWabim, address:145.79.227.34:4938): protocol: rpc: dial tcp 145.79.227.34:4938: i/o timeout; context deadline exceeded Apr 18 02:02:51.267094978 (249h46m1.86120621s) [seg_pos={0 0}, upload=240090]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 02:02:51.371029520 (249h46m1.965140767s) [upload=240090]: Done uploading segments... Apr 18 02:02:51.371072304 (249h46m1.965183546s) [upload=240090]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.106015597s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [.XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X............] duration: 8.77186835s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:02:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.771896292s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX..XX.X] duration: 5.001924801s trace: https://jaeger.dev.storj.io/trace/32eb960997f77b2a FAIL (2026-04-18 02:01:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00194702s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [X.XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X...........] duration: 7.408597945s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:01:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.408705662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.X] duration: 1m0.115852695s trace: https://jaeger.dev.storj.io/trace/7cbe649647c8b3c1 FAIL (2026-04-18 02:01:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 69]: Begin upload segment... Apr 18 02:00:36.761701228 (249h43m47.355812473s) [seg_pos={0 0}, upload=240069]: Done begin upload segment. Apr 18 02:00:36.761796580 (249h43m47.355907833s) [noise=true , address=79.127.205.233:4151 , piece_id=FN2CFC53NWBZ7V2GFJJQUJK2B2CFDM5ZVNXTQG5IQ6B77COXUFXQ, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.761820952 (249h43m47.3559322s) [noise=true , address=145.79.227.38:4990 , piece_id=IRLURZEAHDOHSUAE2VQEIRLEMV4NLQBFT4I3Q4LKXODYAQUR7SWQ, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.762585085 (249h43m47.356696333s) [noise=true , address=79.127.205.235:4269 , piece_id=YYAYTI7M2Q2LQRL3RRJBWRW7NETTOW5RXNGHN46ON3Y6RDDYN4FQ, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.762710363 (249h43m47.356821615s) [noise=true , address=167.17.72.69:4727 , piece_id=RO2BDAHYKDGPNATLQ2MWYJITEMU5HSSYML42BXMHDWL53ZMBI65Q, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.763248995 (249h43m47.357360248s) [noise=true , address=79.127.205.234:4207 , piece_id=G7SJGCIR2UPI33XMOJ2DGDQWCIFSZ2NETWNY7W2SWQEMR2PAXWWQ, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.763288210 (249h43m47.357399464s) [noise=true , address=145.79.227.22:4539 , piece_id=3AIQFFWF22L4ULF33A7WOFTEFPPNN6G2QKPDFMOQCFPR3Q4YES2A, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.763777924 (249h43m47.357889175s) [noise=true , address=193.149.164.71:4299 , piece_id=2FOELHIB2354622LZO3GIYDVFPYZ42ZIGXJG5MCK7A73LJUYR5MA, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.763907709 (249h43m47.35801896s) [noise=true , address=167.17.72.70:4773 , piece_id=RGOG4YLWWPCX3KNU46DVROEYSXSWWKWQW4M3H2FNHSC2R5SEZTTA, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.764266243 (249h43m47.358377491s) [noise=true , address=145.79.227.31:4830 , piece_id=V3WWN74OAREJNCJEEGBGECTCGQJC3E2CJRVT3C4OUYRAKLA4GKQQ, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.764627779 (249h43m47.358739031s) [noise=true , address=145.79.227.51:4424 , piece_id=35VZ2GZOZOI2NPOCVB2ZNTRXVX6RBPDSEQFIB7PN4LDXGYMHHL2A, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.765362741 (249h43m47.359473992s) [noise=true , address=193.149.164.66:4996 , piece_id=RVFXATSE43MVZWB7LICVRYDXC3YBOQDBU3ZL4AXDPDAKUDQNYPYQ, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.765424774 (249h43m47.359536022s) [noise=true , address=145.79.227.11:4745 , piece_id=PH5VFZKOZAX2R6RGWWVFZP4D52RENVZ5E3WST5K4TMSDPCWQZHNQ, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.765964152 (249h43m47.3600754s) [noise=true , address=77.90.31.17:4780 , piece_id=EWXJFXN2X4DXALTMGOHECZRTIC5G62LOIVK2XUGTIZXSUFNWNUJQ, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.766007267 (249h43m47.360118518s) [noise=true , address=167.17.72.68:4640 , piece_id=DGF6KQ6HBQIUP6JKDOHOJJCNSF7DM4XBDRLJTFVOM52EGD2JZZOQ, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.766662178 (249h43m47.360773426s) [noise=true , address=79.127.205.237:4243 , piece_id=5EVJLTJD2TJYZQX4VHP3KEMQDTI455OPXT5TAHOEKQJF34MFOGQQ, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.767322556 (249h43m47.361433809s) [noise=true , address=145.79.227.49:4347 , piece_id=FD4AL4UNEUEAX6MPMMYWRFIE2GTMP6IIN7Y5FBGEGEAWPJJ74SFQ, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.767929627 (249h43m47.362040883s) [noise=true , address=145.79.227.4:4200 , piece_id=UPOVKKDRMOEEEASIFXUT7UOARH3ZPGDESU3R4HKQYHASJV33DJBQ, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.768539767 (249h43m47.362650998s) [noise=true , address=145.79.227.44:4224 , piece_id=AMVHM323KBMO3QJJC2VPS2D4GR5KH2JDPUB36SXDJHPBT3JFJ4DQ, seg_pos={0 0}, upload=240069]: Uploading piece... Apr 18 02:00:36.918143693 (249h43m47.512254944s) [noise=true , address=167.17.72.68:4640 , piece_id=DGF6KQ6HBQIUP6JKDOHOJJCNSF7DM4XBDRLJTFVOM52EGD2JZZOQ, seg_pos={0 0}, upload=240069]: Done uploading piece. err: ecclient: upload failed (node:1X1An1SwLyzf72pHcN7pYsuVchBWbVWJvbNSm8v3cFcgwzS3P1, address:167.17.72.68:4640): protocol: write tcp 139.59.144.22:33766->167.17.72.68:4640: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:00:37.337462175 (249h43m47.93157342s) [upload=240069]: Next returned nil segment Apr 18 02:00:37.337482909 (249h43m47.93159416s) [upload=240069]: Waiting for error group managing segments... Apr 18 02:00:37.533210959 (249h43m48.127322209s) [noise=true , address=145.79.227.31:4830 , piece_id=V3WWN74OAREJNCJEEGBGECTCGQJC3E2CJRVT3C4OUYRAKLA4GKQQ, seg_pos={0 0}, upload=240069]: Done uploading piece. err: <nil> Apr 18 02:00:38.070039939 (249h43m48.664151184s) [noise=true , address=145.79.227.22:4539 , piece_id=3AIQFFWF22L4ULF33A7WOFTEFPPNN6G2QKPDFMOQCFPR3Q4YES2A, seg_pos={0 0}, upload=240069]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:145.79.227.22:4539): protocol: storage node overloaded, request limit: 2000 Apr 18 02:00:38.099924661 (249h43m48.694035913s) [noise=true , address=193.149.164.66:4996 , piece_id=RVFXATSE43MVZWB7LICVRYDXC3YBOQDBU3ZL4AXDPDAKUDQNYPYQ, seg_pos={0 0}, upload=240069]: Done uploading piece. err: <nil> Apr 18 02:00:38.131697909 (249h43m48.725809161s) [noise=true , address=79.127.205.233:4151 , piece_id=FN2CFC53NWBZ7V2GFJJQUJK2B2CFDM5ZVNXTQG5IQ6B77COXUFXQ, seg_pos={0 0}, upload=240069]: Done uploading piece. err: <nil> Apr 18 02:00:38.147098412 (249h43m48.741209662s) [noise=true , address=193.149.164.71:4299 , piece_id=2FOELHIB2354622LZO3GIYDVFPYZ42ZIGXJG5MCK7A73LJUYR5MA, seg_pos={0 0}, upload=240069]: Done uploading piece. err: <nil> Apr 18 02:00:38.170867364 (249h43m48.764978609s) [noise=true , address=79.127.205.234:4207 , piece_id=G7SJGCIR2UPI33XMOJ2DGDQWCIFSZ2NETWNY7W2SWQEMR2PAXWWQ, seg_pos={0 0}, upload=240069]: Done uploading piece. err: <nil> Apr 18 02:00:38.191186605 (249h43m48.785297859s) [noise=true , address=79.127.205.237:4243 , piece_id=5EVJLTJD2TJYZQX4VHP3KEMQDTI455OPXT5TAHOEKQJF34MFOGQQ, seg_pos={0 0}, upload=240069]: Done uploading piece. err: <nil> Apr 18 02:00:38.193744076 (249h43m48.787855322s) [noise=true , address=79.127.205.235:4269 , piece_id=YYAYTI7M2Q2LQRL3RRJBWRW7NETTOW5RXNGHN46ON3Y6RDDYN4FQ, seg_pos={0 0}, upload=240069]: Done uploading piece. err: <nil> Apr 18 02:00:38.372867809 (249h43m48.966979059s) [noise=true , address=145.79.227.51:4424 , piece_id=35VZ2GZOZOI2NPOCVB2ZNTRXVX6RBPDSEQFIB7PN4LDXGYMHHL2A, seg_pos={0 0}, upload=240069]: Done uploading piece. err: ecclient: upload failed (node:12LwZixdSPnCm3Wm27PBY79jj52sjkNh2VuQexagieTGBp4Ps2k, address:145.79.227.51:4424): protocol: storage node overloaded, request limit: 2000 Apr 18 02:00:39.980142127 (249h43m50.57425337s) [noise=true , address=145.79.227.4:4200 , piece_id=UPOVKKDRMOEEEASIFXUT7UOARH3ZPGDESU3R4HKQYHASJV33DJBQ, seg_pos={0 0}, upload=240069]: Done uploading piece. err: ecclient: upload failed (node:1gHRcUu9yyykre3GEugvZMWgdEmDMqr7Fz8HoEpVkA1wuyzzsW, address:145.79.227.4:4200): protocol: storage node overloaded, request limit: 2000 Apr 18 02:00:40.765534931 (249h43m51.35964618s) [noise=true , address=77.90.31.17:4780 , piece_id=EWXJFXN2X4DXALTMGOHECZRTIC5G62LOIVK2XUGTIZXSUFNWNUJQ, seg_pos={0 0}, upload=240069]: Done uploading piece. err: <nil> Apr 18 02:00:44.212022877 (249h43m54.806134123s) [noise=true , address=167.17.72.70:4773 , piece_id=RGOG4YLWWPCX3KNU46DVROEYSXSWWKWQW4M3H2FNHSC2R5SEZTTA, seg_pos={0 0}, upload=240069]: Done uploading piece. err: <nil> Apr 18 02:00:47.292927019 (249h43m57.887038259s) [noise=true , address=167.17.72.69:4727 , piece_id=RO2BDAHYKDGPNATLQ2MWYJITEMU5HSSYML42BXMHDWL53ZMBI65Q, seg_pos={0 0}, upload=240069]: Done uploading piece. err: <nil> Apr 18 02:01:36.351266660 (249h44m46.945377911s) [noise=true , address=145.79.227.49:4347 , piece_id=FD4AL4UNEUEAX6MPMMYWRFIE2GTMP6IIN7Y5FBGEGEAWPJJ74SFQ, seg_pos={0 0}, upload=240069]: Done uploading piece. err: ecclient: upload failed (node:1CbCAPrrYNvhv8129X18ftZmajktekzBFMu5rMLYpHriJbN5PM, address:145.79.227.49:4347): protocol: rpc: dial tcp 145.79.227.49:4347: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:01:36.351554417 (249h44m46.945665667s) [noise=true , address=145.79.227.11:4745 , piece_id=PH5VFZKOZAX2R6RGWWVFZP4D52RENVZ5E3WST5K4TMSDPCWQZHNQ, seg_pos={0 0}, upload=240069]: Done uploading piece. err: ecclient: upload failed (node:1H9sLaFPDPZxKvwXho76wrqPiFe9yu87Ue9BT4ip92qfLmjbnF, address:145.79.227.11:4745): protocol: rpc: dial tcp 145.79.227.11:4745: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:01:36.351788946 (249h44m46.945900198s) [noise=true , address=145.79.227.44:4224 , piece_id=AMVHM323KBMO3QJJC2VPS2D4GR5KH2JDPUB36SXDJHPBT3JFJ4DQ, seg_pos={0 0}, upload=240069]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:145.79.227.44:4224): protocol: rpc: dial tcp 145.79.227.44:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:01:36.352863496 (249h44m46.94697475s) [noise=true , address=145.79.227.38:4990 , piece_id=IRLURZEAHDOHSUAE2VQEIRLEMV4NLQBFT4I3Q4LKXODYAQUR7SWQ, seg_pos={0 0}, upload=240069]: Done uploading piece. err: ecclient: upload failed (node:1gDzCKcMaLTisHw68GYkMbcMJF7rbKnUt48etS8Q7KeiXedbU4, address:145.79.227.38:4990): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:01:36.352908988 (249h44m46.947020236s) [seg_pos={0 0}, upload=240069]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Apr 18 02:01:36.463878996 (249h44m47.05799025s) [upload=240069]: Done uploading segments... Apr 18 02:01:36.463926192 (249h44m47.05803745s) [upload=240069]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.115870457s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [XX.XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X..........] duration: 5.185526762s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 02:00:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.185556954s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX..XX.] duration: 2.603746589s trace: https://jaeger.dev.storj.io/trace/4c93161e44b16e84 FAIL (2026-04-18 02:00:52 UTC) orbiter warning: context deadline exceeded Duration: 2.603765227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.] duration: 39.869025568s trace: https://jaeger.dev.storj.io/trace/77b648841cbf701f FAIL (2026-04-18 02:00:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Uploading segments... Apr 18 01:58:56.343168739 (249h42m6.937279989s) [upload=240040]: Got next segment. Inline: false Apr 18 01:58:56.343210335 (249h42m6.937321577s) [upload=240040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 01:58:56.343218215 (249h42m6.937329469s) [upload=240040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 01:58:56.453163316 (249h42m7.047274558s) [seg_pos={0 0}, upload=240040]: Begin upload segment... Apr 18 01:58:56.453251083 (249h42m7.047362333s) [seg_pos={0 0}, upload=240040]: Done begin upload segment. Apr 18 01:58:56.453298424 (249h42m7.047409669s) [noise=true , address=145.79.227.4:4200 , piece_id=EELZRNCH2GIQCEGPURJS76CUIKVO6UQEJ4BQGOTI7CLRCEG55INQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.453955862 (249h42m7.048067108s) [noise=true , address=145.79.227.41:4119 , piece_id=QZJRD366C6HMWDWJOJFHL4OYN64BJK2DM3XWEPCYY77B5EWIMEYA, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.454427374 (249h42m7.048538605s) [noise=true , address=145.79.227.22:4539 , piece_id=TPLNXMBWFU3M5MF6TIQBN7HO7TCBW3GVDXKPYK7JCI57O7JHUVXQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.454782670 (249h42m7.048893912s) [noise=true , address=145.79.227.11:4740 , piece_id=DJTNDSR6DVNA2LICQX62DO6DMSEMO6JHKPCLKL6WEGL5W27SHZVA, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.455178676 (249h42m7.049289924s) [noise=true , address=167.17.72.69:4714 , piece_id=Y7TWWKT6VEFZ43JXEGUAPAYEUOPT5CI65TWJFMZVZH5YD2ATMHVQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.457071983 (249h42m7.051183231s) [noise=true , address=145.79.227.40:4064 , piece_id=FZENB5O7IENE4B3WNYQM4E545VZVXSEWD3W7Y3J22723I4N6DRFQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.457239881 (249h42m7.051351127s) [noise=true , address=77.90.31.17:4775 , piece_id=RCRQQEPAVVQC2REUI2KKBR7ZCB327PGZM7JMOUI6ROMGO72ZBFVA, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.459570796 (249h42m7.053682044s) [noise=true , address=79.127.205.233:4153 , piece_id=GAJQAWJ6OVMOERRAXWCLP5WUJVAL5ZYJ25HFGSC6TENJYDQFPB3Q, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.461166145 (249h42m7.05527739s) [noise=true , address=145.79.227.13:4817 , piece_id=22DUHYUXWQL66MFT5GDMXXFL6FUICRHYNZ3H54S3BXNBACCMJLLQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.461864476 (249h42m7.055975719s) [noise=true , address=145.79.227.31:4818 , piece_id=6WW2UXKZA5K7HV5SJJ3LSJCLLWBHSGLPUO42C4KIHC2ZQYOTCFLQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.468278368 (249h42m7.062389616s) [noise=true , address=145.79.227.49:4368 , piece_id=PYTD6VYQENWQGF5BR4EKFRUSBMOPZIAJDN3MJNGXVW6NSLVDELEQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.472554092 (249h42m7.066665328s) [noise=true , address=167.17.72.71:4807 , piece_id=AFKHV4YO7DM4QYCSISNCK7CJ5I24TCUXAIGZWWTTMA74OOLH4NCA, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.474249136 (249h42m7.068360383s) [noise=true , address=167.17.72.68:4656 , piece_id=6EPVRLWMVRKVVNFLKNLIQFFUBNRBLEQ7BTFWK6R5TJWUZMNVH52A, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.478700063 (249h42m7.072811305s) [noise=true , address=145.79.227.34:4939 , piece_id=73SWK6BZJOHEVRI4O6BMRW2R6F5NEN7RLVSRDQ2CMNEUQWLWTRUQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.480100510 (249h42m7.074211752s) [noise=true , address=145.79.227.32:4858 , piece_id=W2T5SEPQ63MJLSLMQXM47WEHPIS57TRMIONM32K6XPTBITUEW3SQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.480948041 (249h42m7.075059292s) [noise=true , address=167.17.72.70:4773 , piece_id=L3XDJW3SPNW3AYGPZ36Y4MBUEHIVZF7EGMFHFHPW2IUVGDAJK4GA, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.482193979 (249h42m7.076305218s) [noise=true , address=145.79.227.43:4182 , piece_id=E26QE7CIZ5DKLQ77IAFFI7XVONIOWECA55U5XYBDK52AWDSZVCKQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.483212801 (249h42m7.077324046s) [noise=true , address=79.127.205.234:4191 , piece_id=VPAL3D7NLFTEDRIEXR3YZF3NGTQ3HDWZFBHXN27M6D6TUEEBXGBA, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.552792198 (249h42m7.146903459s) [noise=true , address=145.79.227.31:4818 , piece_id=6WW2UXKZA5K7HV5SJJ3LSJCLLWBHSGLPUO42C4KIHC2ZQYOTCFLQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:58:57.056490950 (249h42m7.650602193s) [upload=240040]: Next returned nil segment Apr 18 01:58:57.056509605 (249h42m7.650620859s) [upload=240040]: Waiting for error group managing segments... Apr 18 01:58:57.179419814 (249h42m7.773531055s) [noise=true , address=145.79.227.4:4200 , piece_id=EELZRNCH2GIQCEGPURJS76CUIKVO6UQEJ4BQGOTI7CLRCEG55INQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:58:57.237739912 (249h42m7.831851164s) [noise=true , address=145.79.227.43:4182 , piece_id=E26QE7CIZ5DKLQ77IAFFI7XVONIOWECA55U5XYBDK52AWDSZVCKQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:58:57.358580237 (249h42m7.95269148s) [noise=true , address=145.79.227.41:4119 , piece_id=QZJRD366C6HMWDWJOJFHL4OYN64BJK2DM3XWEPCYY77B5EWIMEYA, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:58:57.728601279 (249h42m8.32271253s) [noise=true , address=79.127.205.234:4191 , piece_id=VPAL3D7NLFTEDRIEXR3YZF3NGTQ3HDWZFBHXN27M6D6TUEEBXGBA, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:58:57.928771194 (249h42m8.522882424s) [noise=true , address=79.127.205.233:4153 , piece_id=GAJQAWJ6OVMOERRAXWCLP5WUJVAL5ZYJ25HFGSC6TENJYDQFPB3Q, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:58:58.411144431 (249h42m9.005255674s) [noise=true , address=167.17.72.68:4656 , piece_id=6EPVRLWMVRKVVNFLKNLIQFFUBNRBLEQ7BTFWK6R5TJWUZMNVH52A, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:58:59.647253034 (249h42m10.241364271s) [noise=true , address=145.79.227.40:4064 , piece_id=FZENB5O7IENE4B3WNYQM4E545VZVXSEWD3W7Y3J22723I4N6DRFQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:00.319015958 (249h42m10.913127203s) [noise=true , address=145.79.227.22:4539 , piece_id=TPLNXMBWFU3M5MF6TIQBN7HO7TCBW3GVDXKPYK7JCI57O7JHUVXQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:03.212775938 (249h42m13.806887163s) [noise=true , address=77.90.31.17:4775 , piece_id=RCRQQEPAVVQC2REUI2KKBR7ZCB327PGZM7JMOUI6ROMGO72ZBFVA, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:04.163920992 (249h42m14.758032234s) [noise=true , address=167.17.72.70:4773 , piece_id=L3XDJW3SPNW3AYGPZ36Y4MBUEHIVZF7EGMFHFHPW2IUVGDAJK4GA, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:07.463149775 (249h42m18.057261018s) [noise=true , address=167.17.72.71:4807 , piece_id=AFKHV4YO7DM4QYCSISNCK7CJ5I24TCUXAIGZWWTTMA74OOLH4NCA, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:15.942031885 (249h42m26.536143134s) [noise=true , address=145.79.227.32:4858 , piece_id=W2T5SEPQ63MJLSLMQXM47WEHPIS57TRMIONM32K6XPTBITUEW3SQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:18.306950995 (249h42m28.901062239s) [noise=true , address=145.79.227.13:4817 , piece_id=22DUHYUXWQL66MFT5GDMXXFL6FUICRHYNZ3H54S3BXNBACCMJLLQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:56.056013835 (249h43m6.650125086s) [noise=true , address=145.79.227.34:4939 , piece_id=73SWK6BZJOHEVRI4O6BMRW2R6F5NEN7RLVSRDQ2CMNEUQWLWTRUQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: ecclient: upload failed (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu, address:145.79.227.34:4939): protocol: rpc: dial tcp 145.79.227.34:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:59:56.056530650 (249h43m6.6506419s) [noise=true , address=145.79.227.49:4368 , piece_id=PYTD6VYQENWQGF5BR4EKFRUSBMOPZIAJDN3MJNGXVW6NSLVDELEQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: ecclient: upload failed (node:12Q6suyRuFmW1rG5qq8faCFbCkfAPaNfYQaBv6qhL1DNavUffkB, address:145.79.227.49:4368): protocol: rpc: dial tcp 145.79.227.49:4368: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:59:56.056871062 (249h43m6.650982319s) [noise=true , address=167.17.72.69:4714 , piece_id=Y7TWWKT6VEFZ43JXEGUAPAYEUOPT5CI65TWJFMZVZH5YD2ATMHVQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: ecclient: upload failed (node:1nCMopffycySa67FjyZX7oPC3KuzjnbnQPdEnNNN3zcH2YKab6, address:167.17.72.69:4714): protocol: write tcp 139.59.144.22:42704->167.17.72.69:4714: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:59:56.056911415 (249h43m6.651022648s) [noise=true , address=145.79.227.11:4740 , piece_id=DJTNDSR6DVNA2LICQX62DO6DMSEMO6JHKPCLKL6WEGL5W27SHZVA, seg_pos={0 0}, upload=240040]: Done uploading piece. err: ecclient: upload failed (node:12wgQDbE17gP8c6kbKnNyywSkY8Hk7mchcMaymLBQzPwRsSB8QE, address:145.79.227.11:4740): protocol: rpc: dial tcp 145.79.227.11:4740: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:59:56.056965400 (249h43m6.65107664s) [seg_pos={0 0}, upload=240040]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 01:59:56.159779974 (249h43m6.753891221s) [upload=240040]: Done uploading segments... Apr 18 01:59:56.159841104 (249h43m6.753952355s) [upload=240040]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 39.869048531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX] duration: 1m0.106568018s trace: https://jaeger.dev.storj.io/trace/77b648841cbf701f FAIL (2026-04-18 01:59:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Uploading segments... Apr 18 01:58:56.343168739 (249h42m6.937279989s) [upload=240040]: Got next segment. Inline: false Apr 18 01:58:56.343210335 (249h42m6.937321577s) [upload=240040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 01:58:56.343218215 (249h42m6.937329469s) [upload=240040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 01:58:56.453163316 (249h42m7.047274558s) [seg_pos={0 0}, upload=240040]: Begin upload segment... Apr 18 01:58:56.453251083 (249h42m7.047362333s) [seg_pos={0 0}, upload=240040]: Done begin upload segment. Apr 18 01:58:56.453298424 (249h42m7.047409669s) [noise=true , address=145.79.227.4:4200 , piece_id=EELZRNCH2GIQCEGPURJS76CUIKVO6UQEJ4BQGOTI7CLRCEG55INQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.453955862 (249h42m7.048067108s) [noise=true , address=145.79.227.41:4119 , piece_id=QZJRD366C6HMWDWJOJFHL4OYN64BJK2DM3XWEPCYY77B5EWIMEYA, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.454427374 (249h42m7.048538605s) [noise=true , address=145.79.227.22:4539 , piece_id=TPLNXMBWFU3M5MF6TIQBN7HO7TCBW3GVDXKPYK7JCI57O7JHUVXQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.454782670 (249h42m7.048893912s) [noise=true , address=145.79.227.11:4740 , piece_id=DJTNDSR6DVNA2LICQX62DO6DMSEMO6JHKPCLKL6WEGL5W27SHZVA, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.455178676 (249h42m7.049289924s) [noise=true , address=167.17.72.69:4714 , piece_id=Y7TWWKT6VEFZ43JXEGUAPAYEUOPT5CI65TWJFMZVZH5YD2ATMHVQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.457071983 (249h42m7.051183231s) [noise=true , address=145.79.227.40:4064 , piece_id=FZENB5O7IENE4B3WNYQM4E545VZVXSEWD3W7Y3J22723I4N6DRFQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.457239881 (249h42m7.051351127s) [noise=true , address=77.90.31.17:4775 , piece_id=RCRQQEPAVVQC2REUI2KKBR7ZCB327PGZM7JMOUI6ROMGO72ZBFVA, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.459570796 (249h42m7.053682044s) [noise=true , address=79.127.205.233:4153 , piece_id=GAJQAWJ6OVMOERRAXWCLP5WUJVAL5ZYJ25HFGSC6TENJYDQFPB3Q, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.461166145 (249h42m7.05527739s) [noise=true , address=145.79.227.13:4817 , piece_id=22DUHYUXWQL66MFT5GDMXXFL6FUICRHYNZ3H54S3BXNBACCMJLLQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.461864476 (249h42m7.055975719s) [noise=true , address=145.79.227.31:4818 , piece_id=6WW2UXKZA5K7HV5SJJ3LSJCLLWBHSGLPUO42C4KIHC2ZQYOTCFLQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.468278368 (249h42m7.062389616s) [noise=true , address=145.79.227.49:4368 , piece_id=PYTD6VYQENWQGF5BR4EKFRUSBMOPZIAJDN3MJNGXVW6NSLVDELEQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.472554092 (249h42m7.066665328s) [noise=true , address=167.17.72.71:4807 , piece_id=AFKHV4YO7DM4QYCSISNCK7CJ5I24TCUXAIGZWWTTMA74OOLH4NCA, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.474249136 (249h42m7.068360383s) [noise=true , address=167.17.72.68:4656 , piece_id=6EPVRLWMVRKVVNFLKNLIQFFUBNRBLEQ7BTFWK6R5TJWUZMNVH52A, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.478700063 (249h42m7.072811305s) [noise=true , address=145.79.227.34:4939 , piece_id=73SWK6BZJOHEVRI4O6BMRW2R6F5NEN7RLVSRDQ2CMNEUQWLWTRUQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.480100510 (249h42m7.074211752s) [noise=true , address=145.79.227.32:4858 , piece_id=W2T5SEPQ63MJLSLMQXM47WEHPIS57TRMIONM32K6XPTBITUEW3SQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.480948041 (249h42m7.075059292s) [noise=true , address=167.17.72.70:4773 , piece_id=L3XDJW3SPNW3AYGPZ36Y4MBUEHIVZF7EGMFHFHPW2IUVGDAJK4GA, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.482193979 (249h42m7.076305218s) [noise=true , address=145.79.227.43:4182 , piece_id=E26QE7CIZ5DKLQ77IAFFI7XVONIOWECA55U5XYBDK52AWDSZVCKQ, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.483212801 (249h42m7.077324046s) [noise=true , address=79.127.205.234:4191 , piece_id=VPAL3D7NLFTEDRIEXR3YZF3NGTQ3HDWZFBHXN27M6D6TUEEBXGBA, seg_pos={0 0}, upload=240040]: Uploading piece... Apr 18 01:58:56.552792198 (249h42m7.146903459s) [noise=true , address=145.79.227.31:4818 , piece_id=6WW2UXKZA5K7HV5SJJ3LSJCLLWBHSGLPUO42C4KIHC2ZQYOTCFLQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:58:57.056490950 (249h42m7.650602193s) [upload=240040]: Next returned nil segment Apr 18 01:58:57.056509605 (249h42m7.650620859s) [upload=240040]: Waiting for error group managing segments... Apr 18 01:58:57.179419814 (249h42m7.773531055s) [noise=true , address=145.79.227.4:4200 , piece_id=EELZRNCH2GIQCEGPURJS76CUIKVO6UQEJ4BQGOTI7CLRCEG55INQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:58:57.237739912 (249h42m7.831851164s) [noise=true , address=145.79.227.43:4182 , piece_id=E26QE7CIZ5DKLQ77IAFFI7XVONIOWECA55U5XYBDK52AWDSZVCKQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:58:57.358580237 (249h42m7.95269148s) [noise=true , address=145.79.227.41:4119 , piece_id=QZJRD366C6HMWDWJOJFHL4OYN64BJK2DM3XWEPCYY77B5EWIMEYA, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:58:57.728601279 (249h42m8.32271253s) [noise=true , address=79.127.205.234:4191 , piece_id=VPAL3D7NLFTEDRIEXR3YZF3NGTQ3HDWZFBHXN27M6D6TUEEBXGBA, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:58:57.928771194 (249h42m8.522882424s) [noise=true , address=79.127.205.233:4153 , piece_id=GAJQAWJ6OVMOERRAXWCLP5WUJVAL5ZYJ25HFGSC6TENJYDQFPB3Q, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:58:58.411144431 (249h42m9.005255674s) [noise=true , address=167.17.72.68:4656 , piece_id=6EPVRLWMVRKVVNFLKNLIQFFUBNRBLEQ7BTFWK6R5TJWUZMNVH52A, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:58:59.647253034 (249h42m10.241364271s) [noise=true , address=145.79.227.40:4064 , piece_id=FZENB5O7IENE4B3WNYQM4E545VZVXSEWD3W7Y3J22723I4N6DRFQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:00.319015958 (249h42m10.913127203s) [noise=true , address=145.79.227.22:4539 , piece_id=TPLNXMBWFU3M5MF6TIQBN7HO7TCBW3GVDXKPYK7JCI57O7JHUVXQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:03.212775938 (249h42m13.806887163s) [noise=true , address=77.90.31.17:4775 , piece_id=RCRQQEPAVVQC2REUI2KKBR7ZCB327PGZM7JMOUI6ROMGO72ZBFVA, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:04.163920992 (249h42m14.758032234s) [noise=true , address=167.17.72.70:4773 , piece_id=L3XDJW3SPNW3AYGPZ36Y4MBUEHIVZF7EGMFHFHPW2IUVGDAJK4GA, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:07.463149775 (249h42m18.057261018s) [noise=true , address=167.17.72.71:4807 , piece_id=AFKHV4YO7DM4QYCSISNCK7CJ5I24TCUXAIGZWWTTMA74OOLH4NCA, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:15.942031885 (249h42m26.536143134s) [noise=true , address=145.79.227.32:4858 , piece_id=W2T5SEPQ63MJLSLMQXM47WEHPIS57TRMIONM32K6XPTBITUEW3SQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:18.306950995 (249h42m28.901062239s) [noise=true , address=145.79.227.13:4817 , piece_id=22DUHYUXWQL66MFT5GDMXXFL6FUICRHYNZ3H54S3BXNBACCMJLLQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: <nil> Apr 18 01:59:56.056013835 (249h43m6.650125086s) [noise=true , address=145.79.227.34:4939 , piece_id=73SWK6BZJOHEVRI4O6BMRW2R6F5NEN7RLVSRDQ2CMNEUQWLWTRUQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: ecclient: upload failed (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu, address:145.79.227.34:4939): protocol: rpc: dial tcp 145.79.227.34:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:59:56.056530650 (249h43m6.6506419s) [noise=true , address=145.79.227.49:4368 , piece_id=PYTD6VYQENWQGF5BR4EKFRUSBMOPZIAJDN3MJNGXVW6NSLVDELEQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: ecclient: upload failed (node:12Q6suyRuFmW1rG5qq8faCFbCkfAPaNfYQaBv6qhL1DNavUffkB, address:145.79.227.49:4368): protocol: rpc: dial tcp 145.79.227.49:4368: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:59:56.056871062 (249h43m6.650982319s) [noise=true , address=167.17.72.69:4714 , piece_id=Y7TWWKT6VEFZ43JXEGUAPAYEUOPT5CI65TWJFMZVZH5YD2ATMHVQ, seg_pos={0 0}, upload=240040]: Done uploading piece. err: ecclient: upload failed (node:1nCMopffycySa67FjyZX7oPC3KuzjnbnQPdEnNNN3zcH2YKab6, address:167.17.72.69:4714): protocol: write tcp 139.59.144.22:42704->167.17.72.69:4714: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:59:56.056911415 (249h43m6.651022648s) [noise=true , address=145.79.227.11:4740 , piece_id=DJTNDSR6DVNA2LICQX62DO6DMSEMO6JHKPCLKL6WEGL5W27SHZVA, seg_pos={0 0}, upload=240040]: Done uploading piece. err: ecclient: upload failed (node:12wgQDbE17gP8c6kbKnNyywSkY8Hk7mchcMaymLBQzPwRsSB8QE, address:145.79.227.11:4740): protocol: rpc: dial tcp 145.79.227.11:4740: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:59:56.056965400 (249h43m6.65107664s) [seg_pos={0 0}, upload=240040]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 01:59:56.159779974 (249h43m6.753891221s) [upload=240040]: Done uploading segments... Apr 18 01:59:56.159841104 (249h43m6.753952355s) [upload=240040]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.106606265s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [XXX.XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X.........] duration: 10.947773314s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 01:59:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.94780197s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX..XX] duration: 5.000887612s trace: https://jaeger.dev.storj.io/trace/4c93161e44b16e84 FAIL (2026-04-18 01:59:55 UTC) orbiter warning: context deadline exceeded Duration: 5.000912162s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX..X] duration: 5.001407002s trace: https://jaeger.dev.storj.io/trace/3896c8fd9396705c FAIL (2026-04-18 01:59:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001537017s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [.XXX.XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X........] duration: 3.237013056s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 01:59:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.237041724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXX] duration: 1m0.108352093s trace: https://jaeger.dev.storj.io/trace/501c6bd4266c3807 FAIL (2026-04-18 01:58:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== t segment. Inline: false Apr 18 01:57:56.072617445 (249h41m6.666728696s) [upload=240028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 01:57:56.072624288 (249h41m6.666735536s) [upload=240028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 01:57:56.183741089 (249h41m6.77785233s) [seg_pos={0 0}, upload=240028]: Begin upload segment... Apr 18 01:57:56.183818693 (249h41m6.777929921s) [seg_pos={0 0}, upload=240028]: Done begin upload segment. Apr 18 01:57:56.183876848 (249h41m6.777988098s) [noise=true , address=79.127.205.234:4208 , piece_id=AFNLSEOYGVZJ2UDP4VKE4MYQKDJOKKWS47OTMZ7CSBW6AKSVBWNQ, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.183891416 (249h41m6.77800266s) [noise=true , address=145.79.227.49:4372 , piece_id=PBVQJ6CMSTYXMALP3GTAJWQWE7YBC5QBVC22YMVFTZW3VQM4IVMQ, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.184740106 (249h41m6.778851348s) [noise=true , address=79.127.205.237:4237 , piece_id=LYDN3MTQQHQ4QARE44JOIGUC2TRY4ZLRQXIZB6KWRX4ENTMWQE3Q, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.184851102 (249h41m6.778962352s) [noise=true , address=145.79.227.40:4062 , piece_id=BM7K3K7SCH3CU2LQY4DYCSZKUDNEYSLAWXDQN2RPF46R2FVRRLMQ, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.185335721 (249h41m6.779446972s) [noise=true , address=167.17.72.70:4754 , piece_id=II5DLMPHUCAQGPC4B2RDJH2TIA2UGZKRQAFBKOBROIX6BWK7W3OQ, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.185350962 (249h41m6.779462207s) [noise=true , address=45.91.116.13:4178 , piece_id=MA4E2F4XUD7QYAGTKLT4KUTRR2EA5ZDVEKP3B6SA2HVEHSZR3ZQQ, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.185722813 (249h41m6.779834056s) [noise=true , address=193.149.164.71:4300 , piece_id=SY2E76QPVH5Q6LNQODGNVKMRF6VRE24FAF2OX2B6EXCEBGIIYS7A, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.185770655 (249h41m6.779881905s) [noise=true , address=145.79.227.34:4916 , piece_id=X5MUEIOPMIP2PKDNO42C55PVYWUY3LUCZ22YDFR3ZTTVZFT66PEQ, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.186197607 (249h41m6.780308851s) [noise=true , address=145.79.227.7:4320 , piece_id=KZOH7O7D2HRYRNG5XFMDA2GL3H3N4MXONATQK76ELHLCIKSPVIKQ, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.186363936 (249h41m6.780475186s) [noise=true , address=167.17.72.71:4783 , piece_id=7JCNKRFQ3RXEKHDKU32BHOGUJDYKM4VVIQH6ZY22FPCCII7K4AWA, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.186562018 (249h41m6.780673258s) [noise=true , address=193.149.164.66:4992 , piece_id=VA46SOZXHVBLK3C6JPVRV4T4DA5M37XKWIKCXLWZ76ALJ3LUL7KQ, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.186762050 (249h41m6.780873298s) [noise=true , address=77.90.31.19:4870 , piece_id=KVHAXFSSH56A23X4QKPOP6PARMTVK42QU4HOCPTBARLP6YT3TWZA, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.186911188 (249h41m6.781022432s) [noise=true , address=145.79.227.33:4893 , piece_id=VRADZTMHKDMCDGBAXDCRCDSHPRMKXPYHZNYXRCN3SWDCPIKNEOWQ, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.187280343 (249h41m6.781391589s) [noise=true , address=167.17.72.68:4661 , piece_id=IXQEIAFA22Q5C3QHKJQ3LW5JCS6K33ZLQHDWG7VZXEOP3NFKHEPA, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.187905947 (249h41m6.78201719s) [noise=true , address=145.79.227.22:4523 , piece_id=ASFVFYWDWKRVPY5EIBOM6G6QL3W4BAQLHSLE7ECGC7SKUFTLXFEA, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.188303423 (249h41m6.782414668s) [noise=true , address=79.127.205.233:4174 , piece_id=JXUB2FQU657DWR55T2TQCYREU4MAR6X3O7OQCTZRDOMBJ3RM3LJQ, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.188708895 (249h41m6.782820145s) [noise=true , address=79.127.205.235:4255 , piece_id=THZZXCTL24MZNGTVPRFWK5QYPT3TSEDSV7QTJEP5YJIBN5IOJVQA, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.189123203 (249h41m6.783234436s) [noise=true , address=145.79.227.11:4736 , piece_id=ICNW6BNFAGYLQQRTSRXYXQJNPMOM5CAWAVBZKZS5YJMOX6H23YHA, seg_pos={0 0}, upload=240028]: Uploading piece... Apr 18 01:57:56.274884534 (249h41m6.868995783s) [noise=true , address=145.79.227.49:4372 , piece_id=PBVQJ6CMSTYXMALP3GTAJWQWE7YBC5QBVC22YMVFTZW3VQM4IVMQ, seg_pos={0 0}, upload=240028]: Done uploading piece. err: ecclient: upload failed (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3, address:145.79.227.49:4372): protocol: rpc: dial tcp 145.79.227.49:4372: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:57:56.343774409 (249h41m6.93788565s) [noise=true , address=167.17.72.71:4783 , piece_id=7JCNKRFQ3RXEKHDKU32BHOGUJDYKM4VVIQH6ZY22FPCCII7K4AWA, seg_pos={0 0}, upload=240028]: Done uploading piece. err: ecclient: upload failed (node:1kDMS1cpidjeWH8YxBhQTUXn9jCRZ4MQnvAMgUEXPWnYs4jDx8, address:167.17.72.71:4783): protocol: write tcp 139.59.144.22:41530->167.17.72.71:4783: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:57:56.749078414 (249h41m7.343189671s) [upload=240028]: Next returned nil segment Apr 18 01:57:56.749096376 (249h41m7.343207632s) [upload=240028]: Waiting for error group managing segments... Apr 18 01:57:56.888876493 (249h41m7.482987741s) [noise=true , address=145.79.227.7:4320 , piece_id=KZOH7O7D2HRYRNG5XFMDA2GL3H3N4MXONATQK76ELHLCIKSPVIKQ, seg_pos={0 0}, upload=240028]: Done uploading piece. err: <nil> Apr 18 01:57:57.014840614 (249h41m7.608951857s) [noise=true , address=145.79.227.40:4062 , piece_id=BM7K3K7SCH3CU2LQY4DYCSZKUDNEYSLAWXDQN2RPF46R2FVRRLMQ, seg_pos={0 0}, upload=240028]: Done uploading piece. err: <nil> Apr 18 01:57:57.521020174 (249h41m8.115131414s) [noise=true , address=79.127.205.234:4208 , piece_id=AFNLSEOYGVZJ2UDP4VKE4MYQKDJOKKWS47OTMZ7CSBW6AKSVBWNQ, seg_pos={0 0}, upload=240028]: Done uploading piece. err: <nil> Apr 18 01:57:57.557132318 (249h41m8.151243558s) [noise=true , address=79.127.205.237:4237 , piece_id=LYDN3MTQQHQ4QARE44JOIGUC2TRY4ZLRQXIZB6KWRX4ENTMWQE3Q, seg_pos={0 0}, upload=240028]: Done uploading piece. err: <nil> Apr 18 01:57:57.622999008 (249h41m8.217110248s) [noise=true , address=79.127.205.235:4255 , piece_id=THZZXCTL24MZNGTVPRFWK5QYPT3TSEDSV7QTJEP5YJIBN5IOJVQA, seg_pos={0 0}, upload=240028]: Done uploading piece. err: <nil> Apr 18 01:57:57.663860813 (249h41m8.257972052s) [noise=true , address=79.127.205.233:4174 , piece_id=JXUB2FQU657DWR55T2TQCYREU4MAR6X3O7OQCTZRDOMBJ3RM3LJQ, seg_pos={0 0}, upload=240028]: Done uploading piece. err: <nil> Apr 18 01:57:57.673872348 (249h41m8.267983592s) [noise=true , address=167.17.72.68:4661 , piece_id=IXQEIAFA22Q5C3QHKJQ3LW5JCS6K33ZLQHDWG7VZXEOP3NFKHEPA, seg_pos={0 0}, upload=240028]: Done uploading piece. err: <nil> Apr 18 01:57:57.878697504 (249h41m8.472808755s) [noise=true , address=45.91.116.13:4178 , piece_id=MA4E2F4XUD7QYAGTKLT4KUTRR2EA5ZDVEKP3B6SA2HVEHSZR3ZQQ, seg_pos={0 0}, upload=240028]: Done uploading piece. err: ecclient: upload failed (node:125m3sbLAzXdKnjK9WZmasPccheZX63eDagUXqxVCPiakA9ApNX, address:45.91.116.13:4178): protocol: storage node overloaded, request limit: 2000 Apr 18 01:57:59.973540645 (249h41m10.567651886s) [noise=true , address=77.90.31.19:4870 , piece_id=KVHAXFSSH56A23X4QKPOP6PARMTVK42QU4HOCPTBARLP6YT3TWZA, seg_pos={0 0}, upload=240028]: Done uploading piece. err: <nil> Apr 18 01:58:00.245200496 (249h41m10.839311756s) [noise=true , address=167.17.72.70:4754 , piece_id=II5DLMPHUCAQGPC4B2RDJH2TIA2UGZKRQAFBKOBROIX6BWK7W3OQ, seg_pos={0 0}, upload=240028]: Done uploading piece. err: <nil> Apr 18 01:58:01.838650238 (249h41m12.432761481s) [noise=true , address=145.79.227.22:4523 , piece_id=ASFVFYWDWKRVPY5EIBOM6G6QL3W4BAQLHSLE7ECGC7SKUFTLXFEA, seg_pos={0 0}, upload=240028]: Done uploading piece. err: <nil> Apr 18 01:58:02.259555562 (249h41m12.853666813s) [noise=true , address=193.149.164.66:4992 , piece_id=VA46SOZXHVBLK3C6JPVRV4T4DA5M37XKWIKCXLWZ76ALJ3LUL7KQ, seg_pos={0 0}, upload=240028]: Done uploading piece. err: <nil> Apr 18 01:58:02.881060477 (249h41m13.475171744s) [noise=true , address=193.149.164.71:4300 , piece_id=SY2E76QPVH5Q6LNQODGNVKMRF6VRE24FAF2OX2B6EXCEBGIIYS7A, seg_pos={0 0}, upload=240028]: Done uploading piece. err: <nil> Apr 18 01:58:55.785466661 (249h42m6.379577913s) [noise=true , address=145.79.227.34:4916 , piece_id=X5MUEIOPMIP2PKDNO42C55PVYWUY3LUCZ22YDFR3ZTTVZFT66PEQ, seg_pos={0 0}, upload=240028]: Done uploading piece. err: ecclient: upload failed (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa, address:145.79.227.34:4916): protocol: rpc: dial tcp 145.79.227.34:4916: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:58:55.785527998 (249h42m6.379639246s) [noise=true , address=145.79.227.11:4736 , piece_id=ICNW6BNFAGYLQQRTSRXYXQJNPMOM5CAWAVBZKZS5YJMOX6H23YHA, seg_pos={0 0}, upload=240028]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:145.79.227.11:4736): protocol: rpc: dial tcp 145.79.227.11:4736: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:58:55.785781242 (249h42m6.379892491s) [noise=true , address=145.79.227.33:4893 , piece_id=VRADZTMHKDMCDGBAXDCRCDSHPRMKXPYHZNYXRCN3SWDCPIKNEOWQ, seg_pos={0 0}, upload=240028]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:145.79.227.33:4893): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:58:55.785837126 (249h42m6.379948363s) [seg_pos={0 0}, upload=240028]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 18 01:58:55.891847294 (249h42m6.485958533s) [upload=240028]: Done uploading segments... Apr 18 01:58:55.891897261 (249h42m6.486008506s) [upload=240028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.108389001s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [X.XXX.XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X.......] duration: 7.896835582s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 01:58:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.89687986s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX..] duration: 3.014208472s trace: https://jaeger.dev.storj.io/trace/32f5c24d72b51965 FAIL (2026-04-18 01:58:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.014232581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXX] duration: 1m0.107152949s trace: https://jaeger.dev.storj.io/trace/38f1788fb45ca66a FAIL (2026-04-18 01:57:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 91576 (249h40m6.477902811s) [noise=true , address=79.127.205.233:4141 , piece_id=J2N5XCTPHG37XZZYKBSMKUM3RLXBLDZ5TKW5GDAPVKMCYLFZWR7Q, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.885216594 (249h40m6.479327821s) [noise=true , address=79.127.205.235:4284 , piece_id=T4CH45IW3UDJSH62ZLANUCZKUQYMBPYAFMMBJNKDFFJQVALXNSQQ, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.887012405 (249h40m6.481123638s) [noise=true , address=145.79.227.33:4878 , piece_id=RS33H3KBFVGWVVQODJDCKREZU4FMCDB4MCJUJYQPRMGDUBHRUYVQ, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.888637848 (249h40m6.482749079s) [noise=true , address=79.127.205.234:4205 , piece_id=BU75G32OFFI4Y5TAFHIIIKLCNPXW4DRDZXQWUFXQXKMSGM6OVZMA, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.889507510 (249h40m6.483618738s) [noise=true , address=145.79.227.32:4864 , piece_id=VYG3CPVJ4NUJQYU6CFF4LIM2HPSLWIJF52275LR6ARNW63IF5TPA, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.891716441 (249h40m6.485827675s) [noise=true , address=167.17.72.68:4645 , piece_id=JQUCYZATZXMRMPKULT6J2LPRC5QNJYTM6T7DK7TFH642D4VQ3WJA, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.893982457 (249h40m6.488093686s) [noise=true , address=167.17.72.69:4734 , piece_id=GQANUXTNZPXPMF65AAFQDRFPQIH6J25PEBO2VZV4Q4V34WQLKWOA, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.894745796 (249h40m6.48885703s) [noise=true , address=77.90.31.10:4523 , piece_id=64J6OKLRJ57CYIP25KKE5RTUZ4FJRPPQIK2T7TN2B3POHWSPSXDQ, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.895083101 (249h40m6.489194334s) [noise=true , address=145.79.227.49:4351 , piece_id=TWMZ3D3EQVLNB3V6ZF2VGFUKH57EEVGT3JN3OIGMETTUQRKO76LA, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.895683662 (249h40m6.489794896s) [noise=true , address=145.79.227.13:4820 , piece_id=LWDHTBZX62Q52I4ITLN3CEJO7GXXBDANZICCR54IURO47Q3BSLOQ, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.896187221 (249h40m6.490298454s) [noise=true , address=145.79.227.44:4205 , piece_id=W3E7E57OVRNSYW7MUHFOQJTBNYNSVLTUUX5SUEWGCOPZRSDXABMQ, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.896787435 (249h40m6.490898667s) [noise=true , address=167.17.72.71:4811 , piece_id=YB5GHBSRPNN6WBYMWE43BDSINNXW3JHF6QYZ77FFT7OYOML5AOYQ, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.897539265 (249h40m6.491650499s) [noise=true , address=193.149.164.66:4986 , piece_id=NKMIVPEWYKJP3XNXAKNJPB3RYIMAAHRB2VSLZMPB7XR526NJMFHQ, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.898926555 (249h40m6.493037789s) [noise=true , address=77.90.31.19:4870 , piece_id=KS5XK2CCRMFLAUJIG6R7H6PU2JRBU5FTTLGTX3I3AGSCDPV6OXZA, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.901011361 (249h40m6.495122593s) [noise=true , address=145.79.227.11:4755 , piece_id=JWB2JFOGJW3WV6ETK6UWPSFWAHXRR4Y3COWG2Z37PM6RG6CFGK5A, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.903192739 (249h40m6.497303973s) [noise=true , address=145.79.227.38:4998 , piece_id=OBN7I4E4PDG57QBBVO3NYVJ6OPOJ3IYNLGGQP2RTYLGKT3INJRZQ, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.905407409 (249h40m6.499518638s) [noise=true , address=167.17.72.70:4759 , piece_id=Y5JC4BYWCM2A76NBNJAVUXEJXC3KMGVHJZKVN3DD6DT3OPNKVB5Q, seg_pos={0 0}, upload=240016]: Uploading piece... Apr 18 01:56:55.995730315 (249h40m6.589841566s) [noise=true , address=145.79.227.38:4998 , piece_id=OBN7I4E4PDG57QBBVO3NYVJ6OPOJ3IYNLGGQP2RTYLGKT3INJRZQ, seg_pos={0 0}, upload=240016]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:56:56.499457972 (249h40m7.093569202s) [upload=240016]: Next returned nil segment Apr 18 01:56:56.499473473 (249h40m7.09358472s) [upload=240016]: Waiting for error group managing segments... Apr 18 01:56:56.649085533 (249h40m7.243196772s) [noise=true , address=77.90.31.19:4870 , piece_id=KS5XK2CCRMFLAUJIG6R7H6PU2JRBU5FTTLGTX3I3AGSCDPV6OXZA, seg_pos={0 0}, upload=240016]: Done uploading piece. err: ecclient: upload failed (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR, address:77.90.31.19:4870): protocol: storage node overloaded, request limit: 2000 Apr 18 01:56:57.124617450 (249h40m7.718728705s) [noise=true , address=193.149.164.71:4301 , piece_id=2SBAOC46QG6H4SDMPX7TG33Z342PDJRZAS5E4MQEMNHTNNUUDZUQ, seg_pos={0 0}, upload=240016]: Done uploading piece. err: <nil> Apr 18 01:56:57.136719413 (249h40m7.730830659s) [noise=true , address=79.127.205.235:4284 , piece_id=T4CH45IW3UDJSH62ZLANUCZKUQYMBPYAFMMBJNKDFFJQVALXNSQQ, seg_pos={0 0}, upload=240016]: Done uploading piece. err: <nil> Apr 18 01:56:57.247107580 (249h40m7.84121882s) [noise=true , address=77.90.31.10:4523 , piece_id=64J6OKLRJ57CYIP25KKE5RTUZ4FJRPPQIK2T7TN2B3POHWSPSXDQ, seg_pos={0 0}, upload=240016]: Done uploading piece. err: <nil> Apr 18 01:56:57.270452557 (249h40m7.864563805s) [noise=true , address=79.127.205.234:4205 , piece_id=BU75G32OFFI4Y5TAFHIIIKLCNPXW4DRDZXQWUFXQXKMSGM6OVZMA, seg_pos={0 0}, upload=240016]: Done uploading piece. err: <nil> Apr 18 01:56:57.295607378 (249h40m7.889718635s) [noise=true , address=79.127.205.233:4141 , piece_id=J2N5XCTPHG37XZZYKBSMKUM3RLXBLDZ5TKW5GDAPVKMCYLFZWR7Q, seg_pos={0 0}, upload=240016]: Done uploading piece. err: <nil> Apr 18 01:57:02.512071358 (249h40m13.106182608s) [noise=true , address=167.17.72.70:4759 , piece_id=Y5JC4BYWCM2A76NBNJAVUXEJXC3KMGVHJZKVN3DD6DT3OPNKVB5Q, seg_pos={0 0}, upload=240016]: Done uploading piece. err: <nil> Apr 18 01:57:04.624830718 (249h40m15.218941959s) [noise=true , address=167.17.72.71:4811 , piece_id=YB5GHBSRPNN6WBYMWE43BDSINNXW3JHF6QYZ77FFT7OYOML5AOYQ, seg_pos={0 0}, upload=240016]: Done uploading piece. err: <nil> Apr 18 01:57:04.638292278 (249h40m15.232403528s) [noise=true , address=145.79.227.32:4864 , piece_id=VYG3CPVJ4NUJQYU6CFF4LIM2HPSLWIJF52275LR6ARNW63IF5TPA, seg_pos={0 0}, upload=240016]: Done uploading piece. err: <nil> Apr 18 01:57:35.520561097 (249h40m46.114672337s) [noise=true , address=145.79.227.13:4820 , piece_id=LWDHTBZX62Q52I4ITLN3CEJO7GXXBDANZICCR54IURO47Q3BSLOQ, seg_pos={0 0}, upload=240016]: Done uploading piece. err: <nil> Apr 18 01:57:55.505783566 (249h41m6.099894838s) [noise=true , address=193.149.164.66:4986 , piece_id=NKMIVPEWYKJP3XNXAKNJPB3RYIMAAHRB2VSLZMPB7XR526NJMFHQ, seg_pos={0 0}, upload=240016]: Done uploading piece. err: ecclient: upload failed (node:1QdArU4mX4EE2P1fvbP8CeiWEpaxyqL3e55HMtC6uwVW7Xqth9, address:193.149.164.66:4986): protocol: rpc: dial tcp 193.149.164.66:4986: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:57:55.506116666 (249h41m6.100227918s) [noise=true , address=145.79.227.33:4878 , piece_id=RS33H3KBFVGWVVQODJDCKREZU4FMCDB4MCJUJYQPRMGDUBHRUYVQ, seg_pos={0 0}, upload=240016]: Done uploading piece. err: ecclient: upload failed (node:1wX9xRmyv3KWVyJhFmgBt9uYqDU6U768dqHagKZeKBSEnFup5y, address:145.79.227.33:4878): protocol: read tcp 139.59.144.22:44462->145.79.227.33:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:57:55.506277335 (249h41m6.100388583s) [noise=true , address=167.17.72.69:4734 , piece_id=GQANUXTNZPXPMF65AAFQDRFPQIH6J25PEBO2VZV4Q4V34WQLKWOA, seg_pos={0 0}, upload=240016]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: write tcp 139.59.144.22:54104->167.17.72.69:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:57:55.506499488 (249h41m6.100610741s) [noise=true , address=167.17.72.68:4645 , piece_id=JQUCYZATZXMRMPKULT6J2LPRC5QNJYTM6T7DK7TFH642D4VQ3WJA, seg_pos={0 0}, upload=240016]: Done uploading piece. err: ecclient: upload failed (node:12LkJ3iu6FoMMpqkwtLrD9Q5zTrwk7s57bPtJzRw1kgzXTih6kE, address:167.17.72.68:4645): protocol: write tcp 139.59.144.22:54800->167.17.72.68:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:57:55.506531306 (249h41m6.100642553s) [noise=true , address=145.79.227.44:4205 , piece_id=W3E7E57OVRNSYW7MUHFOQJTBNYNSVLTUUX5SUEWGCOPZRSDXABMQ, seg_pos={0 0}, upload=240016]: Done uploading piece. err: ecclient: upload failed (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ, address:145.79.227.44:4205): protocol: rpc: dial tcp 145.79.227.44:4205: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:57:55.506647263 (249h41m6.100758515s) [noise=true , address=145.79.227.49:4351 , piece_id=TWMZ3D3EQVLNB3V6ZF2VGFUKH57EEVGT3JN3OIGMETTUQRKO76LA, seg_pos={0 0}, upload=240016]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:145.79.227.49:4351): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:57:55.506771898 (249h41m6.100883172s) [noise=true , address=145.79.227.11:4755 , piece_id=JWB2JFOGJW3WV6ETK6UWPSFWAHXRR4Y3COWG2Z37PM6RG6CFGK5A, seg_pos={0 0}, upload=240016]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:145.79.227.11:4755): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:57:55.506832547 (249h41m6.1009438s) [seg_pos={0 0}, upload=240016]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Apr 18 01:57:55.609024790 (249h41m6.203136039s) [upload=240016]: Done uploading segments... Apr 18 01:57:55.609094296 (249h41m6.203205546s) [upload=240016]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.107245393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [XX.XXX.XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X......] duration: 3.243157055s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 01:57:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.243280138s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [.XX.XXX.XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X.....] duration: 4.187621276s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 01:57:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.18771482s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.] duration: 2.399409687s trace: https://jaeger.dev.storj.io/trace/32f5c24d72b51965 FAIL (2026-04-18 01:57:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.399440358s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XX] duration: 1m0.103686279s trace: https://jaeger.dev.storj.io/trace/5f7e1088a0ed2cf1 FAIL (2026-04-18 01:56:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== : pb.BatchRequestItem_SegmentBegin Apr 18 01:55:47.667663146 (249h38m58.261774376s) [seg_pos={0 0}, upload=239997]: Begin upload segment... Apr 18 01:55:47.667730626 (249h38m58.261841866s) [seg_pos={0 0}, upload=239997]: Done begin upload segment. Apr 18 01:55:47.667765935 (249h38m58.261877178s) [noise=true , address=77.90.31.17:4777 , piece_id=RM6XVD5MKSDUSG6LG3QTSNDD57YKN7CMLVOKRYPBT7OZORNUDOHQ, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.667880945 (249h38m58.26199219s) [noise=true , address=45.91.116.13:4178 , piece_id=Y4KGXXJQUVX3O4CKOEWQLL6CTJMCMORKXOSN6RJU5XGWWW2D7QDQ, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.668511376 (249h38m58.262622621s) [noise=true , address=64.49.13.137:4495 , piece_id=GSVIPIIMWG37N5ALL3FT6TTLAFYPJFDL7NFMHILD3LGWWX55KK7A, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.669113801 (249h38m58.263225049s) [noise=true , address=193.149.164.143:4694 , piece_id=XUWOK6GOXOSHAYXO5BSB7ZX47L2EGBUWG52S3KEP3YPRW6FFSTKQ, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.670186380 (249h38m58.264297626s) [noise=true , address=145.79.227.35:4503 , piece_id=SP3QNOJLWO4QMJ2CNHH6VMHBXQUCVVEJ3KVUHDGSGH4P3YHQUCEA, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.670687113 (249h38m58.264798363s) [noise=true , address=145.79.227.38:4985 , piece_id=QT5OSXYPW6WB7UK466Q2KQM7OGUCJS4GJ34X65SB4HBZH3WSCXCA, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.671158939 (249h38m58.265270182s) [noise=true , address=167.17.72.69:4730 , piece_id=QPSP73A2P5EGQJAHCVQKAQT5547E33AAY6A4AHWLSOVNRLLPNRWQ, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.671589391 (249h38m58.265700639s) [noise=true , address=167.17.72.68:4645 , piece_id=DZD26B3SN6U4H6V24AGHQDSVYJ7GFZW2FOYP5K2BXI7V4AFYWUCQ, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.672048749 (249h38m58.266159992s) [noise=true , address=145.79.227.22:4539 , piece_id=2KMCWAPCYX635G2LEKL3HBBAY4RYSKKB47IYEBQN26HA2YYKAL5Q, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.672709456 (249h38m58.266820701s) [noise=true , address=193.149.164.66:4974 , piece_id=A6BD6B372HEQE5IYKG64RPPR3M2ZJMCY6GTEX2PVMMBVMTIUPRRA, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.673243810 (249h38m58.26735506s) [noise=true , address=193.149.164.71:4314 , piece_id=DRHXYN4ZCEYS2BERPAUXH5DCLO4RC3YMBDLJ5CRSIUNN5GF2TLQQ, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.673800607 (249h38m58.267911858s) [noise=true , address=145.79.227.11:4750 , piece_id=ZIC23KVC6Q3MAFRPHJWGSAH25WA3JCQCVT4NLE3BNW2ANKUX7GGQ, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.675926854 (249h38m58.270038105s) [noise=true , address=145.79.227.44:4219 , piece_id=WPCOKI7OW3H6YXTCAKBSPVIICZWTAGS3NCNUZ5Y2KGAGKJMXRANQ, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.677755151 (249h38m58.271866401s) [noise=true , address=167.17.72.71:4793 , piece_id=53WHPNZQJ3BJLHACWCT77CARTVCHCJZ2KCR5J5IWRDM4XV2BEGBA, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.679616493 (249h38m58.273727741s) [noise=true , address=79.127.205.235:4268 , piece_id=TGN2PH5GRFIR4J7KO55OMMNIM2WWGPYTZPPY3QFYWCUW7CDJVCSA, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.681430094 (249h38m58.275541342s) [noise=true , address=145.79.227.19:4095 , piece_id=P3TPIKPTD7JSK5P7PVOZUIUNBKQHRXV44WPKZKFXRIHIXHS3QFYA, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.683317783 (249h38m58.277429031s) [noise=true , address=79.127.205.233:4174 , piece_id=3NAU26IYP6P7HWYUMH5HCGL5XUMUTD4GFC5IR3XA4ICNIU4BWHJQ, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.688034841 (249h38m58.282146091s) [noise=true , address=79.127.205.234:4204 , piece_id=E24QYARPYRBGHOJCEF5UTI7OXAKBZG7NSWSA5GIGZIS2E6MSGMEQ, seg_pos={0 0}, upload=239997]: Uploading piece... Apr 18 01:55:47.830076702 (249h38m58.424187953s) [noise=true , address=193.149.164.143:4694 , piece_id=XUWOK6GOXOSHAYXO5BSB7ZX47L2EGBUWG52S3KEP3YPRW6FFSTKQ, seg_pos={0 0}, upload=239997]: Done uploading piece. err: ecclient: upload failed (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, address:193.149.164.143:4694): protocol: write tcp 139.59.144.22:57328->193.149.164.143:4694: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:55:48.235058986 (249h38m58.829170213s) [upload=239997]: Next returned nil segment Apr 18 01:55:48.235078712 (249h38m58.829189961s) [upload=239997]: Waiting for error group managing segments... Apr 18 01:55:48.383123617 (249h38m58.977234846s) [noise=true , address=145.79.227.35:4503 , piece_id=SP3QNOJLWO4QMJ2CNHH6VMHBXQUCVVEJ3KVUHDGSGH4P3YHQUCEA, seg_pos={0 0}, upload=239997]: Done uploading piece. err: <nil> Apr 18 01:55:48.396937799 (249h38m58.991049038s) [noise=true , address=45.91.116.13:4178 , piece_id=Y4KGXXJQUVX3O4CKOEWQLL6CTJMCMORKXOSN6RJU5XGWWW2D7QDQ, seg_pos={0 0}, upload=239997]: Done uploading piece. err: <nil> Apr 18 01:55:48.955900481 (249h38m59.550011729s) [noise=true , address=64.49.13.137:4495 , piece_id=GSVIPIIMWG37N5ALL3FT6TTLAFYPJFDL7NFMHILD3LGWWX55KK7A, seg_pos={0 0}, upload=239997]: Done uploading piece. err: <nil> Apr 18 01:55:49.015027578 (249h38m59.609138817s) [noise=true , address=193.149.164.66:4974 , piece_id=A6BD6B372HEQE5IYKG64RPPR3M2ZJMCY6GTEX2PVMMBVMTIUPRRA, seg_pos={0 0}, upload=239997]: Done uploading piece. err: <nil> Apr 18 01:55:49.085574601 (249h38m59.679685841s) [noise=true , address=79.127.205.234:4204 , piece_id=E24QYARPYRBGHOJCEF5UTI7OXAKBZG7NSWSA5GIGZIS2E6MSGMEQ, seg_pos={0 0}, upload=239997]: Done uploading piece. err: <nil> Apr 18 01:55:49.136458817 (249h38m59.730570058s) [noise=true , address=79.127.205.233:4174 , piece_id=3NAU26IYP6P7HWYUMH5HCGL5XUMUTD4GFC5IR3XA4ICNIU4BWHJQ, seg_pos={0 0}, upload=239997]: Done uploading piece. err: <nil> Apr 18 01:55:49.173756952 (249h38m59.767868195s) [noise=true , address=79.127.205.235:4268 , piece_id=TGN2PH5GRFIR4J7KO55OMMNIM2WWGPYTZPPY3QFYWCUW7CDJVCSA, seg_pos={0 0}, upload=239997]: Done uploading piece. err: <nil> Apr 18 01:55:49.429041929 (249h39m0.023153179s) [noise=true , address=193.149.164.71:4314 , piece_id=DRHXYN4ZCEYS2BERPAUXH5DCLO4RC3YMBDLJ5CRSIUNN5GF2TLQQ, seg_pos={0 0}, upload=239997]: Done uploading piece. err: <nil> Apr 18 01:55:49.571755043 (249h39m0.165866295s) [noise=true , address=145.79.227.19:4095 , piece_id=P3TPIKPTD7JSK5P7PVOZUIUNBKQHRXV44WPKZKFXRIHIXHS3QFYA, seg_pos={0 0}, upload=239997]: Done uploading piece. err: <nil> Apr 18 01:55:49.586708536 (249h39m0.180819788s) [noise=true , address=145.79.227.22:4539 , piece_id=2KMCWAPCYX635G2LEKL3HBBAY4RYSKKB47IYEBQN26HA2YYKAL5Q, seg_pos={0 0}, upload=239997]: Done uploading piece. err: <nil> Apr 18 01:55:49.873836252 (249h39m0.467947503s) [noise=true , address=145.79.227.11:4750 , piece_id=ZIC23KVC6Q3MAFRPHJWGSAH25WA3JCQCVT4NLE3BNW2ANKUX7GGQ, seg_pos={0 0}, upload=239997]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/a2/log-00000000000205a2-00005052: no space left on device; EOF Apr 18 01:55:56.770906178 (249h39m7.365017427s) [noise=true , address=167.17.72.71:4793 , piece_id=53WHPNZQJ3BJLHACWCT77CARTVCHCJZ2KCR5J5IWRDM4XV2BEGBA, seg_pos={0 0}, upload=239997]: Done uploading piece. err: <nil> Apr 18 01:55:56.930470138 (249h39m7.524581378s) [noise=true , address=77.90.31.17:4777 , piece_id=RM6XVD5MKSDUSG6LG3QTSNDD57YKN7CMLVOKRYPBT7OZORNUDOHQ, seg_pos={0 0}, upload=239997]: Done uploading piece. err: <nil> Apr 18 01:56:47.285790541 (249h39m57.879901783s) [noise=true , address=167.17.72.68:4645 , piece_id=DZD26B3SN6U4H6V24AGHQDSVYJ7GFZW2FOYP5K2BXI7V4AFYWUCQ, seg_pos={0 0}, upload=239997]: Done uploading piece. err: ecclient: upload failed (node:12LkJ3iu6FoMMpqkwtLrD9Q5zTrwk7s57bPtJzRw1kgzXTih6kE, address:167.17.72.68:4645): protocol: rpc: dial tcp 167.17.72.68:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:56:47.285978616 (249h39m57.880089867s) [noise=true , address=145.79.227.38:4985 , piece_id=QT5OSXYPW6WB7UK466Q2KQM7OGUCJS4GJ34X65SB4HBZH3WSCXCA, seg_pos={0 0}, upload=239997]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:145.79.227.38:4985): protocol: rpc: dial tcp 145.79.227.38:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:56:47.286119542 (249h39m57.880230791s) [noise=true , address=167.17.72.69:4730 , piece_id=QPSP73A2P5EGQJAHCVQKAQT5547E33AAY6A4AHWLSOVNRLLPNRWQ, seg_pos={0 0}, upload=239997]: Done uploading piece. err: ecclient: upload failed (node:125STJwNfwwMQ8V1nCHxFXrTBr9haCjGwdDVQhG67RwvgQKwaTr, address:167.17.72.69:4730): protocol: rpc: dial tcp 167.17.72.69:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:56:47.286498580 (249h39m57.880609828s) [noise=true , address=145.79.227.44:4219 , piece_id=WPCOKI7OW3H6YXTCAKBSPVIICZWTAGS3NCNUZ5Y2KGAGKJMXRANQ, seg_pos={0 0}, upload=239997]: Done uploading piece. err: ecclient: upload failed (node:12TqSxiG3LYYc3W5T6wqiuuKwoh1LvCdntou7ve6TAEbXYf2QKn, address:145.79.227.44:4219): protocol: rpc: dial tcp 145.79.227.44:4219: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:56:47.286538718 (249h39m57.880649944s) [seg_pos={0 0}, upload=239997]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Apr 18 01:56:47.387161141 (249h39m57.981272381s) [upload=239997]: Done uploading segments... Apr 18 01:56:47.387207268 (249h39m57.981318515s) [upload=239997]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.103703847s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX] duration: 5.000985577s trace: https://jaeger.dev.storj.io/trace/32f5c24d72b51965 FAIL (2026-04-18 01:56:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001023745s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [..XX.XXX.XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X....] duration: 4.475826615s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 01:56:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.475860064s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [...XX.XXX.XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X...] duration: 4.606404986s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 01:56:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.606480574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.X] duration: 1m0.109118558s trace: https://jaeger.dev.storj.io/trace/62633d98f9c5dbe FAIL (2026-04-18 01:55:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m57.8942696s) [upload=239981]: Starting upload Apr 18 01:54:47.300257221 (249h37m57.894368469s) [upload=239981]: Done starting upload Apr 18 01:54:47.300447337 (249h37m57.894558565s) [upload=239981]: Uploading segments... Apr 18 01:54:47.300776985 (249h37m57.894888232s) [upload=239981]: Got next segment. Inline: false Apr 18 01:54:47.300919808 (249h37m57.895031058s) [upload=239981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 01:54:47.300927624 (249h37m57.895038873s) [upload=239981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 01:54:47.412938580 (249h37m58.007049827s) [seg_pos={0 0}, upload=239981]: Begin upload segment... Apr 18 01:54:47.413026873 (249h37m58.007138114s) [seg_pos={0 0}, upload=239981]: Done begin upload segment. Apr 18 01:54:47.413079581 (249h37m58.007190835s) [noise=true , address=167.17.72.69:4730 , piece_id=4SMYCVFBJMFMB557NQUME6BY26XZ5CCM7B37S3WZUGL5KD6NFFOA, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.413097988 (249h37m58.007209238s) [noise=true , address=145.79.227.11:4763 , piece_id=47V7QJ276UPJ4WQL6ZYKQDMQEJADRPKL6ZGEMA56UE3CAW4WKCHA, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.413803421 (249h37m58.007914666s) [noise=true , address=79.127.205.237:4224 , piece_id=BRS43HZ4SPA7FQEWJY6B5WZADU2N7IGWXFKOT72KLAL5ZL4UTKJQ, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.413971874 (249h37m58.008083116s) [noise=true , address=77.90.31.17:4792 , piece_id=ZMD7JLBWYX6JQU3P7RS6MF4SOAPS2E6SGYZHXY7QATP52ONURX4Q, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.414551484 (249h37m58.008662726s) [noise=true , address=167.17.72.71:4784 , piece_id=NYROT5AVMBYUYYEYEARFGGOA2UHG2SS3JATY3UQQPBMK7U6MQZBQ, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.415019867 (249h37m58.009131106s) [noise=true , address=167.17.72.68:4671 , piece_id=JVS4SEJWDHD6HLHBHUPQ2C2CGKNEFUAB6XVU5VBASTYEA3DNLURQ, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.415525532 (249h37m58.009636778s) [noise=true , address=79.127.205.234:4208 , piece_id=A44UWRWGKDCGGJ52OKEUOPYJAWZSZC2NUYDA4BGGGBE2D5WLLMJA, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.416016704 (249h37m58.010127965s) [noise=true , address=167.17.72.70:4748 , piece_id=4IUCBZXRNLI4LXAFTX5OUM367D6CZVXZ4MURVTLUB6SJKQAK2Y7Q, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.416585570 (249h37m58.01069681s) [noise=true , address=145.79.227.32:4842 , piece_id=6QWSNAFLMLH7GMSTYCVA43GTHLH545AZKFDMZTANUVLCL6C3565Q, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.423129717 (249h37m58.017240961s) [noise=true , address=193.149.164.71:4288 , piece_id=VYLLQIYORNIQ6OITIGPML56576VRBJ63MQV4HVT5MIRC5PLSZXSA, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.425947172 (249h37m58.020058424s) [noise=true , address=145.79.227.44:4217 , piece_id=ZHRX5HVVYDBPQF2OC6UYI7G5MVOZ32I3Q3FFVMXZZ2WYUANRO4FA, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.427712815 (249h37m58.021824055s) [noise=true , address=79.127.205.233:4142 , piece_id=W2DJ3O34X63KXQ4VWFX7B3QLZUNRJSRZFPKPYA6GG3NRXQEPV4YQ, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.428968356 (249h37m58.023079606s) [noise=true , address=145.79.227.2:4120 , piece_id=ZFQ5ZSAWZVYOFWMUVIUWLWURTYF3YMSFGYEXZWJL3KHWYSCLYSCA, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.430996198 (249h37m58.025107444s) [noise=true , address=64.49.13.137:4495 , piece_id=HDNMMC5XV67RRWYOZCY4KS7T4574VWKIHTWRLSX3LCH6IZEFNFRA, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.432172549 (249h37m58.0262838s) [noise=true , address=193.149.164.143:4700 , piece_id=CFB4SUHQI6ETRZGNWADD6GD2INCZGWHJJYF3T32Z62ZQ2SNGAVHA, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.433498368 (249h37m58.027609613s) [noise=true , address=79.127.205.235:4275 , piece_id=BAPA35Q57HGINWIWKSCJQHEGMAN62IAN7PMWGL5HBHEBPIKEJUDA, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.434554865 (249h37m58.028666114s) [noise=true , address=145.79.227.34:4936 , piece_id=7NIM6VJHBXQGNXF3IBK4DVPGOUER5XIOHOCGV6SVLU5XV3GERHAA, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.436223504 (249h37m58.03033475s) [noise=true , address=145.79.227.38:4012 , piece_id=Z7ZCUFLF3KBHIK6NJKIKJMWEMMYHXU4VMJYKUILA2RIESDOFG4EA, seg_pos={0 0}, upload=239981]: Uploading piece... Apr 18 01:54:47.997994610 (249h37m58.592105844s) [upload=239981]: Next returned nil segment Apr 18 01:54:47.998014127 (249h37m58.592125379s) [upload=239981]: Waiting for error group managing segments... Apr 18 01:54:48.757473695 (249h37m59.351584943s) [noise=true , address=193.149.164.71:4288 , piece_id=VYLLQIYORNIQ6OITIGPML56576VRBJ63MQV4HVT5MIRC5PLSZXSA, seg_pos={0 0}, upload=239981]: Done uploading piece. err: <nil> Apr 18 01:54:48.806745960 (249h37m59.400857199s) [noise=true , address=79.127.205.234:4208 , piece_id=A44UWRWGKDCGGJ52OKEUOPYJAWZSZC2NUYDA4BGGGBE2D5WLLMJA, seg_pos={0 0}, upload=239981]: Done uploading piece. err: <nil> Apr 18 01:54:48.846509443 (249h37m59.440620686s) [noise=true , address=79.127.205.233:4142 , piece_id=W2DJ3O34X63KXQ4VWFX7B3QLZUNRJSRZFPKPYA6GG3NRXQEPV4YQ, seg_pos={0 0}, upload=239981]: Done uploading piece. err: <nil> Apr 18 01:54:48.864043094 (249h37m59.458154339s) [noise=true , address=79.127.205.235:4275 , piece_id=BAPA35Q57HGINWIWKSCJQHEGMAN62IAN7PMWGL5HBHEBPIKEJUDA, seg_pos={0 0}, upload=239981]: Done uploading piece. err: <nil> Apr 18 01:54:48.923820510 (249h37m59.517931759s) [noise=true , address=79.127.205.237:4224 , piece_id=BRS43HZ4SPA7FQEWJY6B5WZADU2N7IGWXFKOT72KLAL5ZL4UTKJQ, seg_pos={0 0}, upload=239981]: Done uploading piece. err: <nil> Apr 18 01:54:49.938607567 (249h38m0.532718807s) [noise=true , address=64.49.13.137:4495 , piece_id=HDNMMC5XV67RRWYOZCY4KS7T4574VWKIHTWRLSX3LCH6IZEFNFRA, seg_pos={0 0}, upload=239981]: Done uploading piece. err: ecclient: upload failed (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1, address:64.49.13.137:4495): protocol: storage node overloaded, request limit: 2000 Apr 18 01:54:50.054932949 (249h38m0.649044199s) [noise=true , address=145.79.227.11:4763 , piece_id=47V7QJ276UPJ4WQL6ZYKQDMQEJADRPKL6ZGEMA56UE3CAW4WKCHA, seg_pos={0 0}, upload=239981]: Done uploading piece. err: <nil> Apr 18 01:54:52.082316509 (249h38m2.676427766s) [noise=true , address=167.17.72.71:4784 , piece_id=NYROT5AVMBYUYYEYEARFGGOA2UHG2SS3JATY3UQQPBMK7U6MQZBQ, seg_pos={0 0}, upload=239981]: Done uploading piece. err: <nil> Apr 18 01:54:52.288595690 (249h38m2.88270693s) [noise=true , address=167.17.72.68:4671 , piece_id=JVS4SEJWDHD6HLHBHUPQ2C2CGKNEFUAB6XVU5VBASTYEA3DNLURQ, seg_pos={0 0}, upload=239981]: Done uploading piece. err: <nil> Apr 18 01:54:54.480694494 (249h38m5.074805733s) [noise=true , address=193.149.164.143:4700 , piece_id=CFB4SUHQI6ETRZGNWADD6GD2INCZGWHJJYF3T32Z62ZQ2SNGAVHA, seg_pos={0 0}, upload=239981]: Done uploading piece. err: <nil> Apr 18 01:54:55.871235864 (249h38m6.465347108s) [noise=true , address=77.90.31.17:4792 , piece_id=ZMD7JLBWYX6JQU3P7RS6MF4SOAPS2E6SGYZHXY7QATP52ONURX4Q, seg_pos={0 0}, upload=239981]: Done uploading piece. err: <nil> Apr 18 01:55:00.634380210 (249h38m11.228491453s) [noise=true , address=167.17.72.70:4748 , piece_id=4IUCBZXRNLI4LXAFTX5OUM367D6CZVXZ4MURVTLUB6SJKQAK2Y7Q, seg_pos={0 0}, upload=239981]: Done uploading piece. err: <nil> Apr 18 01:55:03.687822175 (249h38m14.281933418s) [noise=true , address=145.79.227.32:4842 , piece_id=6QWSNAFLMLH7GMSTYCVA43GTHLH545AZKFDMZTANUVLCL6C3565Q, seg_pos={0 0}, upload=239981]: Done uploading piece. err: <nil> Apr 18 01:55:06.776205420 (249h38m17.370316667s) [noise=true , address=145.79.227.2:4120 , piece_id=ZFQ5ZSAWZVYOFWMUVIUWLWURTYF3YMSFGYEXZWJL3KHWYSCLYSCA, seg_pos={0 0}, upload=239981]: Done uploading piece. err: <nil> Apr 18 01:55:47.014139021 (249h38m57.608250271s) [noise=true , address=167.17.72.69:4730 , piece_id=4SMYCVFBJMFMB557NQUME6BY26XZ5CCM7B37S3WZUGL5KD6NFFOA, seg_pos={0 0}, upload=239981]: Done uploading piece. err: ecclient: upload failed (node:125STJwNfwwMQ8V1nCHxFXrTBr9haCjGwdDVQhG67RwvgQKwaTr, address:167.17.72.69:4730): protocol: rpc: dial tcp 167.17.72.69:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:55:47.014394328 (249h38m57.608505577s) [noise=true , address=145.79.227.38:4012 , piece_id=Z7ZCUFLF3KBHIK6NJKIKJMWEMMYHXU4VMJYKUILA2RIESDOFG4EA, seg_pos={0 0}, upload=239981]: Done uploading piece. err: ecclient: upload failed (node:12ZS5nRRvwoF6T7AHt5Z83Z4r1eyzjMQo5yt9keVUa1UE3DR1W2, address:145.79.227.38:4012): protocol: rpc: dial tcp 145.79.227.38:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:55:47.014924300 (249h38m57.60903555s) [noise=true , address=145.79.227.34:4936 , piece_id=7NIM6VJHBXQGNXF3IBK4DVPGOUER5XIOHOCGV6SVLU5XV3GERHAA, seg_pos={0 0}, upload=239981]: Done uploading piece. err: ecclient: upload failed (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8, address:145.79.227.34:4936): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:55:47.015598539 (249h38m57.609709786s) [noise=true , address=145.79.227.44:4217 , piece_id=ZHRX5HVVYDBPQF2OC6UYI7G5MVOZ32I3Q3FFVMXZZ2WYUANRO4FA, seg_pos={0 0}, upload=239981]: Done uploading piece. err: ecclient: upload failed (node:12bw9V8UMi7pFxqJ8fh22x5ZPmtJUJyJGw8hD9NhQcfHdfx4cZj, address:145.79.227.44:4217): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:55:47.015633454 (249h38m57.609744695s) [seg_pos={0 0}, upload=239981]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 01:55:47.121418853 (249h38m57.715530095s) [upload=239981]: Done uploading segments... Apr 18 01:55:47.121467843 (249h38m57.715579086s) [upload=239981]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.1091984s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.X] duration: 5.001801779s trace: https://jaeger.dev.storj.io/trace/6187b6b3efc3cdd3 FAIL (2026-04-18 01:55:47 UTC) orbiter warning: context deadline exceeded Duration: 5.00182725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [X...XX.XXX.XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X..] duration: 9.38764542s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 01:55:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.387738217s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.] duration: 3.361544451s trace: https://jaeger.dev.storj.io/trace/28ca7aa27d3631a5 FAIL (2026-04-18 01:55:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.361586063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [XX...XX.XXX.XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X.] duration: 4.428699645s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 01:54:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.428719035s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [.XX...XX.XXX.XXXX..X..X.XX.XX..X.XXX..X.XXXX....X..........X] duration: 1m0.001494741s trace: https://jaeger.dev.storj.io/trace/9a3abeebd2986c0 FAIL (2026-04-18 01:54:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00152827s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.] duration: 34.815649605s trace: https://jaeger.dev.storj.io/trace/6027cff4dfa51d72 FAIL (2026-04-18 01:54:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2:57.446198698 (249h36m8.040309952s) [upload=239951]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 01:52:57.560669645 (249h36m8.154780893s) [seg_pos={0 0}, upload=239951]: Begin upload segment... Apr 18 01:52:57.560755451 (249h36m8.154866686s) [seg_pos={0 0}, upload=239951]: Done begin upload segment. Apr 18 01:52:57.560794534 (249h36m8.154905784s) [noise=true , address=167.17.72.69:4727 , piece_id=5NSWEMUEGCBJZDNYZCJG43DL7NUL6QK2W3BBP32DHT3R56BJC2UQ, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.560815162 (249h36m8.154926414s) [noise=true , address=193.149.164.71:4296 , piece_id=7YUV3TRQGAQNKCSG7T4FYY6ASCLBFAJGHDLUAFZQLUPOBFMGSQ5A, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.560948739 (249h36m8.155059985s) [noise=true , address=145.79.227.43:4176 , piece_id=N5XIOVPKZEBPYHOEMSXQT4GGYI55RY54VQQB7QROYCXF22E53ZRA, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.561492661 (249h36m8.15560391s) [noise=true , address=145.79.227.9:4399 , piece_id=DMHF57OTMLOOV7CY3KFZPL4EOYO3W5E4VEE6UFLUSXDKLA23GEXQ, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.561585172 (249h36m8.155696425s) [noise=true , address=64.49.13.139:4474 , piece_id=KALDOTAV2VBNMXI4452ZZJVLHJ7X7BQ3EO476GNJMHULRNVV6ZPA, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.562131770 (249h36m8.15624302s) [noise=true , address=145.79.227.34:4918 , piece_id=IWTWBLKS6XM3GBYATN7RAWEF73BGF6UQ5HZ7A7PAIFHVHYWXPD5A, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.562630818 (249h36m8.156742079s) [noise=true , address=79.127.205.233:4143 , piece_id=OGYWHGWE2F2TMN2MOMSHLU4UTC36ZSL7FBCQPWLSI4NND5KUMF5A, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.563080731 (249h36m8.157191988s) [noise=true , address=193.149.164.66:4973 , piece_id=SDM3YBKZXVTWZ7Z5VIDN2NJA25FRJSNSVVKAWFZ2336MRNSOJD7A, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.563554715 (249h36m8.157665963s) [noise=true , address=167.17.72.70:4778 , piece_id=5OIO5SGTMBXHNUPNHNG6O4OBH2FQJDSDBQ26Z2UT2WMPX7NGUZQA, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.563999635 (249h36m8.158110884s) [noise=true , address=167.17.72.68:4658 , piece_id=GG5T5QHSUYSSW3DWLB66JMXJUWWE6Z6BWAJAXSJM7D5JIHPBBTNA, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.564474100 (249h36m8.158585351s) [noise=true , address=145.79.227.38:4999 , piece_id=67VAT2QPXF7VU5A3LLKTFZ24FHPUGWVMIBJAUDBMHLMJ2DVIMEMQ, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.566435416 (249h36m8.160546665s) [noise=true , address=167.17.72.71:4786 , piece_id=3LECDUX7EIAH7FD3R7POTZFCQF6OTEMSDATQ5B47PCEKDDOIFETQ, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.566886024 (249h36m8.160997263s) [noise=true , address=145.79.227.21:4481 , piece_id=ZT2G7XP5ZNPDV3AJ424OPWDQBLVNXATNKBEL5TXM2IUDSVPHGW7Q, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.567287216 (249h36m8.161398458s) [noise=true , address=145.79.227.13:4818 , piece_id=7GVXOTW5RDN2YMXUKSHGBMCOPMCR7RD74Q6QYN7L3GNZ2VYRRZKA, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.567653035 (249h36m8.161764277s) [noise=true , address=79.127.205.235:4263 , piece_id=5Y3QRZWKUTNN6M6LQWRX6GTNCIFLLQKGJXJTLYDW6PVFKGXFNVAQ, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.568053773 (249h36m8.162165016s) [noise=true , address=145.79.227.44:4222 , piece_id=WXPUSFPLE2R7TWWVL5QKNHTAIAAUDXRKRTV4G4OOLNORKK7PTK2Q, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.568611458 (249h36m8.162722703s) [noise=true , address=79.127.205.237:4234 , piece_id=EWEMT7LHS2PVY3NRKT3WTBVPCX7CLEROAZXLIGWLVHL3T5IRDQMQ, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:57.570529060 (249h36m8.164640309s) [noise=true , address=145.79.227.11:4748 , piece_id=3B5QVRTPUN4DP66Y3YBVLUVWQF4J42RFDLQSXHEFQGTES4HHGU2A, seg_pos={0 0}, upload=239951]: Uploading piece... Apr 18 01:52:58.129986624 (249h36m8.724097866s) [upload=239951]: Next returned nil segment Apr 18 01:52:58.130004703 (249h36m8.724115954s) [upload=239951]: Waiting for error group managing segments... Apr 18 01:52:58.352571820 (249h36m8.94668306s) [noise=true , address=145.79.227.43:4176 , piece_id=N5XIOVPKZEBPYHOEMSXQT4GGYI55RY54VQQB7QROYCXF22E53ZRA, seg_pos={0 0}, upload=239951]: Done uploading piece. err: <nil> Apr 18 01:52:58.437407537 (249h36m9.031518778s) [noise=true , address=145.79.227.11:4748 , piece_id=3B5QVRTPUN4DP66Y3YBVLUVWQF4J42RFDLQSXHEFQGTES4HHGU2A, seg_pos={0 0}, upload=239951]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: storage node overloaded, request limit: 2000 Apr 18 01:52:58.939476329 (249h36m9.533587558s) [noise=true , address=79.127.205.233:4143 , piece_id=OGYWHGWE2F2TMN2MOMSHLU4UTC36ZSL7FBCQPWLSI4NND5KUMF5A, seg_pos={0 0}, upload=239951]: Done uploading piece. err: <nil> Apr 18 01:52:58.941723343 (249h36m9.535834582s) [noise=true , address=193.149.164.71:4296 , piece_id=7YUV3TRQGAQNKCSG7T4FYY6ASCLBFAJGHDLUAFZQLUPOBFMGSQ5A, seg_pos={0 0}, upload=239951]: Done uploading piece. err: <nil> Apr 18 01:52:59.022106544 (249h36m9.616217796s) [noise=true , address=79.127.205.235:4263 , piece_id=5Y3QRZWKUTNN6M6LQWRX6GTNCIFLLQKGJXJTLYDW6PVFKGXFNVAQ, seg_pos={0 0}, upload=239951]: Done uploading piece. err: <nil> Apr 18 01:52:59.032172984 (249h36m9.62628423s) [noise=true , address=79.127.205.237:4234 , piece_id=EWEMT7LHS2PVY3NRKT3WTBVPCX7CLEROAZXLIGWLVHL3T5IRDQMQ, seg_pos={0 0}, upload=239951]: Done uploading piece. err: <nil> Apr 18 01:52:59.078242546 (249h36m9.672353786s) [noise=true , address=145.79.227.9:4399 , piece_id=DMHF57OTMLOOV7CY3KFZPL4EOYO3W5E4VEE6UFLUSXDKLA23GEXQ, seg_pos={0 0}, upload=239951]: Done uploading piece. err: <nil> Apr 18 01:52:59.463663135 (249h36m10.057774383s) [noise=true , address=64.49.13.139:4474 , piece_id=KALDOTAV2VBNMXI4452ZZJVLHJ7X7BQ3EO476GNJMHULRNVV6ZPA, seg_pos={0 0}, upload=239951]: Done uploading piece. err: <nil> Apr 18 01:53:00.457072758 (249h36m11.051184003s) [noise=true , address=145.79.227.21:4481 , piece_id=ZT2G7XP5ZNPDV3AJ424OPWDQBLVNXATNKBEL5TXM2IUDSVPHGW7Q, seg_pos={0 0}, upload=239951]: Done uploading piece. err: ecclient: upload failed (node:12h7TPnAFgxFKain3dR7H7cqRtmYg66Px2zLWgmRcoq7TpQqpoe, address:145.79.227.21:4481): protocol: storage node overloaded, request limit: 2000 Apr 18 01:53:08.456081473 (249h36m19.050192714s) [noise=true , address=167.17.72.68:4658 , piece_id=GG5T5QHSUYSSW3DWLB66JMXJUWWE6Z6BWAJAXSJM7D5JIHPBBTNA, seg_pos={0 0}, upload=239951]: Done uploading piece. err: <nil> Apr 18 01:53:08.851920511 (249h36m19.446031765s) [noise=true , address=193.149.164.66:4973 , piece_id=SDM3YBKZXVTWZ7Z5VIDN2NJA25FRJSNSVVKAWFZ2336MRNSOJD7A, seg_pos={0 0}, upload=239951]: Done uploading piece. err: <nil> Apr 18 01:53:12.158126592 (249h36m22.752237825s) [noise=true , address=167.17.72.69:4727 , piece_id=5NSWEMUEGCBJZDNYZCJG43DL7NUL6QK2W3BBP32DHT3R56BJC2UQ, seg_pos={0 0}, upload=239951]: Done uploading piece. err: <nil> Apr 18 01:53:16.693045736 (249h36m27.287156975s) [noise=true , address=167.17.72.71:4786 , piece_id=3LECDUX7EIAH7FD3R7POTZFCQF6OTEMSDATQ5B47PCEKDDOIFETQ, seg_pos={0 0}, upload=239951]: Done uploading piece. err: <nil> Apr 18 01:53:57.155316896 (249h37m7.749428141s) [noise=true , address=145.79.227.44:4222 , piece_id=WXPUSFPLE2R7TWWVL5QKNHTAIAAUDXRKRTV4G4OOLNORKK7PTK2Q, seg_pos={0 0}, upload=239951]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: rpc: dial tcp 145.79.227.44:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:53:57.155346551 (249h37m7.7494578s) [noise=true , address=145.79.227.13:4818 , piece_id=7GVXOTW5RDN2YMXUKSHGBMCOPMCR7RD74Q6QYN7L3GNZ2VYRRZKA, seg_pos={0 0}, upload=239951]: Done uploading piece. err: ecclient: upload failed (node:1235eFkz7CFwwiM8UUZU1vnqDneEic7RpFzeTU46gvTMZWhhox8, address:145.79.227.13:4818): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:53:57.155572280 (249h37m7.749683534s) [noise=true , address=167.17.72.70:4778 , piece_id=5OIO5SGTMBXHNUPNHNG6O4OBH2FQJDSDBQ26Z2UT2WMPX7NGUZQA, seg_pos={0 0}, upload=239951]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:54442->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:53:57.155637429 (249h37m7.749748693s) [noise=true , address=145.79.227.34:4918 , piece_id=IWTWBLKS6XM3GBYATN7RAWEF73BGF6UQ5HZ7A7PAIFHVHYWXPD5A, seg_pos={0 0}, upload=239951]: Done uploading piece. err: ecclient: upload failed (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ, address:145.79.227.34:4918): protocol: rpc: dial tcp 145.79.227.34:4918: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:53:57.155738207 (249h37m7.74984946s) [noise=true , address=145.79.227.38:4999 , piece_id=67VAT2QPXF7VU5A3LLKTFZ24FHPUGWVMIBJAUDBMHLMJ2DVIMEMQ, seg_pos={0 0}, upload=239951]: Done uploading piece. err: ecclient: upload failed (node:1tSiyPFy1xn1YRCLmKmPmbTm3LUmDP9nT1YRCXnsuyY6XsLKrG, address:145.79.227.38:4999): protocol: rpc: dial tcp 145.79.227.38:4999: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:53:57.155783309 (249h37m7.749894543s) [seg_pos={0 0}, upload=239951]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Apr 18 01:53:57.260434955 (249h37m7.854546216s) [upload=239951]: Done uploading segments... Apr 18 01:53:57.260492871 (249h37m7.854604111s) [upload=239951]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 34.815673536s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 5.000979315s trace: https://jaeger.dev.storj.io/trace/28ca7aa27d3631a5 FAIL (2026-04-18 01:53:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001088613s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)