Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX.......X................X..........X....................] duration: 4.19138375s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:03:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.191432995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX.......X................X..........X...................] duration: 5.277039435s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:03:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.27708981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX.......X................X..........X..................] duration: 4.372431337s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:02:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.372463143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XX.......X................X..........X.................] duration: 3.376546171s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:02:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.37657943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XX.......X................X..........X................] duration: 3.353140006s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:01:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.3531924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......XX.......X................X..........X...............] duration: 3.395152541s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 03:00:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.395184838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........XX.......X................X..........X..............] duration: 3.295972749s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:59:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.29599074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X........XX.......X................X..........X.............] duration: 3.332573677s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:59:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.332605844s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X........XX.......X................X..........X............] duration: 8.392539838s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:58:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.392570513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X........XX.......X................X..........X...........] duration: 3.751556213s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:58:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.751587308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X........XX.......X................X..........X..........] duration: 3.572690486s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:57:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.572721368s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X........XX.......X................X..........X.........] duration: 4.669417085s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:56:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.669447907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X........XX.......X................X..........X........] duration: 3.453765254s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:55:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.453793049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.....X........XX.......X................X..........X.......] duration: 3.647451503s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:54:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.647480795s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 2.101312565s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:54:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.101343733s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.716836095s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:54:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.716896685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX.....X........XX.......X................X..........X......] duration: 3.077933661s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:53:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.077986768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXX.....X........XX.......X................X..........X.....] duration: 4.444518945s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:53:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.444615283s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 1.588668303s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:53:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.588717526s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X..X........................................................] duration: 2.179223015s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:52:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.179253483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [XX..X.......................................................] duration: 1.691367884s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:52:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.691403599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XXX.....X........XX.......X................X..........X....] duration: 4.266021975s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:52:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.266054812s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.XX..X......................................................] duration: 1.688126664s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:51:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.688157487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.XXX.....X........XX.......X................X..........X...] duration: 3.743293905s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:51:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.743327629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [XX.XXX.....X........XX.......X................X..........X..] duration: 3.617777884s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:51:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.617791193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..XX..X.....................................................] duration: 1.630833487s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:51:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.630859269s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...XX..X....................................................] duration: 2.199049457s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:50:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.199168603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXX.XXX.....X........XX.......X................X..........X.] duration: 3.339505223s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:49:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.339541956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XXX.XXX.....X........XX.......X................X..........X] duration: 1m0.109656501s trace: https://jaeger.dev.storj.io/trace/61d9c64893637c9d FAIL (2026-04-18 02:49:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== m49.952831062s) [upload=240849]: Done starting upload Apr 18 02:48:39.359012897 (250h31m49.953124136s) [upload=240849]: Uploading segments... Apr 18 02:48:39.359058739 (250h31m49.953169978s) [upload=240849]: Got next segment. Inline: false Apr 18 02:48:39.359115483 (250h31m49.953226722s) [upload=240849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:48:39.359119808 (250h31m49.953231047s) [upload=240849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:48:39.471158598 (250h31m50.065269837s) [seg_pos={0 0}, upload=240849]: Begin upload segment... Apr 18 02:48:39.471256498 (250h31m50.065367744s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474033606 (250h31m50.06814488s) [seg_pos={0 0}, upload=240849]: Done begin upload segment. Apr 18 02:48:39.474124310 (250h31m50.068235556s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.474678319 (250h31m50.06878956s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475144100 (250h31m50.069255342s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.475624403 (250h31m50.06973565s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476061714 (250h31m50.070172953s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.476674868 (250h31m50.070786109s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.477998869 (250h31m50.072110113s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.478100399 (250h31m50.072211637s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.479686611 (250h31m50.073797861s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.480037485 (250h31m50.074148718s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481253475 (250h31m50.075364718s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.481787106 (250h31m50.075898351s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.483319044 (250h31m50.077430293s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.485268010 (250h31m50.079379258s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.486548274 (250h31m50.080659522s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.488212913 (250h31m50.082324161s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.489730230 (250h31m50.083841473s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Uploading piece... Apr 18 02:48:39.629494246 (250h31m50.223605497s) [noise=true , address=167.17.72.71:4790 , piece_id=VMPCLPHROBJQE7X6BOA7FMPA4IR6MUND3US3AAY3WA6WWN4IO55Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1X1up5L3UEKavzuZXUcAPAhjPHh3qCtuHx9nCpBQiWzc6KdZ4S, address:167.17.72.71:4790): protocol: write tcp 139.59.144.22:36064->167.17.72.71:4790: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:39.676993335 (250h31m50.271104578s) [noise=true , address=167.17.72.70:4758 , piece_id=RCX6IK7PXH2DZIEFYLAYSNBBUMQGQ3IJZ3EW6F5QAWX63G4R5JGA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12jsp58A4cu3VAvKWtq6C73pmUG3Aeg2Fayf9T7Qeay9NhqPAL1, address:167.17.72.70:4758): protocol: rpc: dial tcp 167.17.72.70:4758: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:48:40.034573526 (250h31m50.628684757s) [upload=240849]: Next returned nil segment Apr 18 02:48:40.034637379 (250h31m50.628748625s) [upload=240849]: Waiting for error group managing segments... Apr 18 02:48:40.209506125 (250h31m50.803617373s) [noise=true , address=145.79.227.23:4569 , piece_id=Y3TMSZYHHZKJ7Y2MMIGJ5H4L6ANHZRLC3RDDZ6633AFGNNNHDJHA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.213325290 (250h31m50.807436533s) [noise=true , address=45.91.116.13:4178 , piece_id=XOTH6XJTTTJMACQJAY6XD5NV5FZH6DYIYCBSPOKTPS5FABDYN4NA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219181504 (250h31m50.813292748s) [noise=true , address=145.79.227.32:4867 , piece_id=67UJN6EA2DAENWLZOQYYJ7DFSDE5J4TCBBWESQOYLNRNCY7DUNBQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.219632897 (250h31m50.813744144s) [noise=true , address=145.79.227.43:4177 , piece_id=NE2IGYAGMTSXV5CSLOL7KOZRSD4JCUT672GW7QPOTPSMNIL5ELTA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.728816515 (250h31m51.322927755s) [noise=true , address=193.149.164.66:4969 , piece_id=EKCZJVFBUPHJT6NYMMWM3X4DSRPAUA4ZCIAJY66PTI7STB44CNHQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.729279266 (250h31m51.323390508s) [noise=true , address=64.49.13.169:4707 , piece_id=J3SLGNDXYEMZXTHGM7OYJZZE6ZV7KIUUEEPGHWAOHCQFKN2VN4SQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.812920008 (250h31m51.407031237s) [noise=true , address=145.79.227.11:4761 , piece_id=BY2BY75TYM7JA6LMHDJ6RNOTSXF272EXDLEFWXBIWH3BUGEIX7XQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.854550980 (250h31m51.448662219s) [noise=true , address=79.127.205.237:4230 , piece_id=5F32OBCAK5EPU3CMA7GJOVDNQYYXNCZZIS45VP2RV4YMNOOAOKRQ, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.874929135 (250h31m51.469040375s) [noise=true , address=79.127.205.233:4160 , piece_id=UYYZNIGSM3MSN5OX4FC55Y5B73C2T52QQJ5B7XZOHLJ6VOZAMBAA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.887406700 (250h31m51.48151795s) [noise=true , address=79.127.205.235:4267 , piece_id=TQKOYDR55CQ3LDCZGX2KYNH3M5K6VCWAI5DYWJ3OKZB6UWX23D4A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:40.907581653 (250h31m51.501692913s) [noise=true , address=79.127.205.234:4212 , piece_id=P22TZXEAGNGL6IF5HWADDDY63X5CDDJJRVTPTYAYZXYV6IIZ6M2A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:42.420390712 (250h31m53.014501941s) [noise=true , address=145.79.227.29:4967 , piece_id=DE7IDP7UAQ6ETA6XVOUVUVJZPBKRB53ZACQMWO2OYRKSLCC3AV7Q, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:48:43.034621343 (250h31m53.628732585s) [noise=true , address=193.149.164.143:4697 , piece_id=6BZLENZJOQJWNZ457ACUXQOWY23MSY5ZS4EDY6ANOELBEQYWH52A, seg_pos={0 0}, upload=240849]: Done uploading piece. err: <nil> Apr 18 02:49:39.076954089 (250h32m49.671065338s) [noise=true , address=145.79.227.38:4986 , piece_id=4YAF2SEOKDZPO2CWOVC7YJRFOA5JSGCLQRUK2MMRLELXHZ37IXSA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:145.79.227.38:4986): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077424773 (250h32m49.671536016s) [noise=true , address=145.79.227.49:4364 , piece_id=5PW7AIQYUEYVJSKMEG6KE3EQYATRT5SWEE5OTVBEZHWNDQAWYKWA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:145.79.227.49:4364): protocol: rpc: dial tcp 145.79.227.49:4364: i/o timeout; context deadline exceeded Apr 18 02:49:39.077648832 (250h32m49.671760081s) [noise=true , address=167.17.72.68:4652 , piece_id=H5SXXVCEHCMLDZQBXOWCI3UESPJOK5UAIBW456HLPE4IXQWOBVKA, seg_pos={0 0}, upload=240849]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:49:39.077687258 (250h32m49.671798486s) [seg_pos={0 0}, upload=240849]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:49:39.183431347 (250h32m49.777542597s) [upload=240849]: Done uploading segments... Apr 18 02:49:39.183516902 (250h32m49.777628153s) [upload=240849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.109687219s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....XX..X...................................................] duration: 1.597039391s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:49:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.597072681s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....XX..X..................................................] duration: 2.290725192s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:48:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.290755909s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X.....XX..X.................................................] duration: 1.701171601s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:47:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.701201386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.XXX.XXX.....X........XX.......X................X..........] duration: 4.997052493s trace: https://jaeger.dev.storj.io/trace/31005e07d94faebc FAIL (2026-04-18 02:47:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 02:39:18.903757261 (250h22m29.4978685s) [upload=240701]: Got next segment. Inline: false Apr 18 02:39:18.903830365 (250h22m29.497941603s) [upload=240701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:39:18.903837205 (250h22m29.497948444s) [upload=240701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:39:19.014891342 (250h22m29.609002572s) [seg_pos={0 0}, upload=240701]: Begin upload segment... Apr 18 02:39:19.014972185 (250h22m29.609083415s) [seg_pos={0 0}, upload=240701]: Done begin upload segment. Apr 18 02:39:19.015019360 (250h22m29.609130603s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015032108 (250h22m29.609143353s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015104542 (250h22m29.609215781s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015691354 (250h22m29.609802603s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.016299539 (250h22m29.610410788s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017036654 (250h22m29.611147898s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017269327 (250h22m29.611380573s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017746091 (250h22m29.611857335s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018291629 (250h22m29.612402878s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018353743 (250h22m29.612464986s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018721265 (250h22m29.612832515s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018839235 (250h22m29.61295048s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019288928 (250h22m29.613400176s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019746809 (250h22m29.613858054s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020163463 (250h22m29.614274708s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020821300 (250h22m29.614932548s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021249261 (250h22m29.615360503s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021618436 (250h22m29.615729677s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.164125748 (250h22m29.758236994s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:60394->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.200099142 (250h22m29.79421039s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34470->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.580941409 (250h22m30.175052661s) [upload=240701]: Next returned nil segment Apr 18 02:39:19.580961207 (250h22m30.175072451s) [upload=240701]: Waiting for error group managing segments... Apr 18 02:39:19.741177031 (250h22m30.33528829s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.745737016 (250h22m30.339848276s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.789115076 (250h22m30.383226333s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.848298928 (250h22m30.442410169s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.199475147 (250h22m30.793586388s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.319894286 (250h22m30.914005536s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.356429032 (250h22m30.950540277s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.383013757 (250h22m30.977125007s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.384162295 (250h22m30.978273543s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.424713162 (250h22m31.018824409s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.439978786 (250h22m31.034090027s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.476434966 (250h22m31.070546206s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.802472404 (250h22m31.396583644s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:40:18.617604639 (250h23m29.211715882s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:53148->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.617816606 (250h23m29.211927855s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.618072575 (250h23m29.212183825s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:145.79.227.49:4363): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:40:18.618108549 (250h23m29.212219783s) [seg_pos={0 0}, upload=240701]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:40:18.722218529 (250h23m29.31632977s) [upload=240701]: Done uploading segments... Apr 18 02:40:18.722256343 (250h23m29.316367586s) [upload=240701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.99714701s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X.....XX..X................................................] duration: 1.845973764s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:47:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.846001724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX.XXX.XXX.....X........XX.......X................X.........] duration: 3.716933881s trace: https://jaeger.dev.storj.io/trace/31005e07d94faebc FAIL (2026-04-18 02:46:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 02:39:18.903757261 (250h22m29.4978685s) [upload=240701]: Got next segment. Inline: false Apr 18 02:39:18.903830365 (250h22m29.497941603s) [upload=240701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:39:18.903837205 (250h22m29.497948444s) [upload=240701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:39:19.014891342 (250h22m29.609002572s) [seg_pos={0 0}, upload=240701]: Begin upload segment... Apr 18 02:39:19.014972185 (250h22m29.609083415s) [seg_pos={0 0}, upload=240701]: Done begin upload segment. Apr 18 02:39:19.015019360 (250h22m29.609130603s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015032108 (250h22m29.609143353s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015104542 (250h22m29.609215781s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015691354 (250h22m29.609802603s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.016299539 (250h22m29.610410788s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017036654 (250h22m29.611147898s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017269327 (250h22m29.611380573s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017746091 (250h22m29.611857335s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018291629 (250h22m29.612402878s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018353743 (250h22m29.612464986s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018721265 (250h22m29.612832515s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018839235 (250h22m29.61295048s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019288928 (250h22m29.613400176s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019746809 (250h22m29.613858054s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020163463 (250h22m29.614274708s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020821300 (250h22m29.614932548s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021249261 (250h22m29.615360503s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021618436 (250h22m29.615729677s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.164125748 (250h22m29.758236994s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:60394->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.200099142 (250h22m29.79421039s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34470->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.580941409 (250h22m30.175052661s) [upload=240701]: Next returned nil segment Apr 18 02:39:19.580961207 (250h22m30.175072451s) [upload=240701]: Waiting for error group managing segments... Apr 18 02:39:19.741177031 (250h22m30.33528829s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.745737016 (250h22m30.339848276s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.789115076 (250h22m30.383226333s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.848298928 (250h22m30.442410169s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.199475147 (250h22m30.793586388s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.319894286 (250h22m30.914005536s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.356429032 (250h22m30.950540277s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.383013757 (250h22m30.977125007s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.384162295 (250h22m30.978273543s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.424713162 (250h22m31.018824409s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.439978786 (250h22m31.034090027s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.476434966 (250h22m31.070546206s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.802472404 (250h22m31.396583644s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:40:18.617604639 (250h23m29.211715882s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:53148->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.617816606 (250h23m29.211927855s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.618072575 (250h23m29.212183825s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:145.79.227.49:4363): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:40:18.618108549 (250h23m29.212219783s) [seg_pos={0 0}, upload=240701]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:40:18.722218529 (250h23m29.31632977s) [upload=240701]: Done uploading segments... Apr 18 02:40:18.722256343 (250h23m29.316367586s) [upload=240701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.716968736s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X.X.....XX..X...............................................] duration: 2.245082656s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:46:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.24518613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXX.XXX.XXX.....X........XX.......X................X........] duration: 3.425615646s trace: https://jaeger.dev.storj.io/trace/31005e07d94faebc FAIL (2026-04-18 02:45:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 02:39:18.903757261 (250h22m29.4978685s) [upload=240701]: Got next segment. Inline: false Apr 18 02:39:18.903830365 (250h22m29.497941603s) [upload=240701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:39:18.903837205 (250h22m29.497948444s) [upload=240701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:39:19.014891342 (250h22m29.609002572s) [seg_pos={0 0}, upload=240701]: Begin upload segment... Apr 18 02:39:19.014972185 (250h22m29.609083415s) [seg_pos={0 0}, upload=240701]: Done begin upload segment. Apr 18 02:39:19.015019360 (250h22m29.609130603s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015032108 (250h22m29.609143353s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015104542 (250h22m29.609215781s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015691354 (250h22m29.609802603s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.016299539 (250h22m29.610410788s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017036654 (250h22m29.611147898s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017269327 (250h22m29.611380573s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017746091 (250h22m29.611857335s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018291629 (250h22m29.612402878s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018353743 (250h22m29.612464986s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018721265 (250h22m29.612832515s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018839235 (250h22m29.61295048s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019288928 (250h22m29.613400176s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019746809 (250h22m29.613858054s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020163463 (250h22m29.614274708s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020821300 (250h22m29.614932548s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021249261 (250h22m29.615360503s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021618436 (250h22m29.615729677s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.164125748 (250h22m29.758236994s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:60394->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.200099142 (250h22m29.79421039s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34470->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.580941409 (250h22m30.175052661s) [upload=240701]: Next returned nil segment Apr 18 02:39:19.580961207 (250h22m30.175072451s) [upload=240701]: Waiting for error group managing segments... Apr 18 02:39:19.741177031 (250h22m30.33528829s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.745737016 (250h22m30.339848276s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.789115076 (250h22m30.383226333s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.848298928 (250h22m30.442410169s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.199475147 (250h22m30.793586388s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.319894286 (250h22m30.914005536s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.356429032 (250h22m30.950540277s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.383013757 (250h22m30.977125007s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.384162295 (250h22m30.978273543s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.424713162 (250h22m31.018824409s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.439978786 (250h22m31.034090027s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.476434966 (250h22m31.070546206s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.802472404 (250h22m31.396583644s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:40:18.617604639 (250h23m29.211715882s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:53148->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.617816606 (250h23m29.211927855s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.618072575 (250h23m29.212183825s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:145.79.227.49:4363): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:40:18.618108549 (250h23m29.212219783s) [seg_pos={0 0}, upload=240701]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:40:18.722218529 (250h23m29.31632977s) [upload=240701]: Done uploading segments... Apr 18 02:40:18.722256343 (250h23m29.316367586s) [upload=240701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.425647854s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [XX.X.....XX..X..............................................] duration: 1.794700472s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:45:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.794732984s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXXX.XXX.XXX.....X........XX.......X................X.......] duration: 3.515629093s trace: https://jaeger.dev.storj.io/trace/31005e07d94faebc FAIL (2026-04-18 02:44:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 02:39:18.903757261 (250h22m29.4978685s) [upload=240701]: Got next segment. Inline: false Apr 18 02:39:18.903830365 (250h22m29.497941603s) [upload=240701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:39:18.903837205 (250h22m29.497948444s) [upload=240701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:39:19.014891342 (250h22m29.609002572s) [seg_pos={0 0}, upload=240701]: Begin upload segment... Apr 18 02:39:19.014972185 (250h22m29.609083415s) [seg_pos={0 0}, upload=240701]: Done begin upload segment. Apr 18 02:39:19.015019360 (250h22m29.609130603s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015032108 (250h22m29.609143353s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015104542 (250h22m29.609215781s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015691354 (250h22m29.609802603s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.016299539 (250h22m29.610410788s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017036654 (250h22m29.611147898s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017269327 (250h22m29.611380573s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017746091 (250h22m29.611857335s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018291629 (250h22m29.612402878s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018353743 (250h22m29.612464986s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018721265 (250h22m29.612832515s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018839235 (250h22m29.61295048s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019288928 (250h22m29.613400176s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019746809 (250h22m29.613858054s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020163463 (250h22m29.614274708s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020821300 (250h22m29.614932548s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021249261 (250h22m29.615360503s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021618436 (250h22m29.615729677s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.164125748 (250h22m29.758236994s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:60394->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.200099142 (250h22m29.79421039s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34470->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.580941409 (250h22m30.175052661s) [upload=240701]: Next returned nil segment Apr 18 02:39:19.580961207 (250h22m30.175072451s) [upload=240701]: Waiting for error group managing segments... Apr 18 02:39:19.741177031 (250h22m30.33528829s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.745737016 (250h22m30.339848276s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.789115076 (250h22m30.383226333s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.848298928 (250h22m30.442410169s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.199475147 (250h22m30.793586388s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.319894286 (250h22m30.914005536s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.356429032 (250h22m30.950540277s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.383013757 (250h22m30.977125007s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.384162295 (250h22m30.978273543s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.424713162 (250h22m31.018824409s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.439978786 (250h22m31.034090027s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.476434966 (250h22m31.070546206s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.802472404 (250h22m31.396583644s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:40:18.617604639 (250h23m29.211715882s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:53148->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.617816606 (250h23m29.211927855s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.618072575 (250h23m29.212183825s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:145.79.227.49:4363): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:40:18.618108549 (250h23m29.212219783s) [seg_pos={0 0}, upload=240701]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:40:18.722218529 (250h23m29.31632977s) [upload=240701]: Done uploading segments... Apr 18 02:40:18.722256343 (250h23m29.316367586s) [upload=240701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.515664546s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.XX.X.....XX..X.............................................] duration: 1.750580662s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:44:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.75061835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXXXX.XXX.XXX.....X........XX.......X................X......] duration: 3.661376269s trace: https://jaeger.dev.storj.io/trace/31005e07d94faebc FAIL (2026-04-18 02:43:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 02:39:18.903757261 (250h22m29.4978685s) [upload=240701]: Got next segment. Inline: false Apr 18 02:39:18.903830365 (250h22m29.497941603s) [upload=240701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:39:18.903837205 (250h22m29.497948444s) [upload=240701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:39:19.014891342 (250h22m29.609002572s) [seg_pos={0 0}, upload=240701]: Begin upload segment... Apr 18 02:39:19.014972185 (250h22m29.609083415s) [seg_pos={0 0}, upload=240701]: Done begin upload segment. Apr 18 02:39:19.015019360 (250h22m29.609130603s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015032108 (250h22m29.609143353s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015104542 (250h22m29.609215781s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015691354 (250h22m29.609802603s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.016299539 (250h22m29.610410788s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017036654 (250h22m29.611147898s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017269327 (250h22m29.611380573s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017746091 (250h22m29.611857335s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018291629 (250h22m29.612402878s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018353743 (250h22m29.612464986s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018721265 (250h22m29.612832515s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018839235 (250h22m29.61295048s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019288928 (250h22m29.613400176s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019746809 (250h22m29.613858054s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020163463 (250h22m29.614274708s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020821300 (250h22m29.614932548s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021249261 (250h22m29.615360503s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021618436 (250h22m29.615729677s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.164125748 (250h22m29.758236994s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:60394->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.200099142 (250h22m29.79421039s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34470->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.580941409 (250h22m30.175052661s) [upload=240701]: Next returned nil segment Apr 18 02:39:19.580961207 (250h22m30.175072451s) [upload=240701]: Waiting for error group managing segments... Apr 18 02:39:19.741177031 (250h22m30.33528829s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.745737016 (250h22m30.339848276s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.789115076 (250h22m30.383226333s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.848298928 (250h22m30.442410169s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.199475147 (250h22m30.793586388s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.319894286 (250h22m30.914005536s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.356429032 (250h22m30.950540277s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.383013757 (250h22m30.977125007s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.384162295 (250h22m30.978273543s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.424713162 (250h22m31.018824409s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.439978786 (250h22m31.034090027s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.476434966 (250h22m31.070546206s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.802472404 (250h22m31.396583644s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:40:18.617604639 (250h23m29.211715882s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:53148->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.617816606 (250h23m29.211927855s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.618072575 (250h23m29.212183825s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:145.79.227.49:4363): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:40:18.618108549 (250h23m29.212219783s) [seg_pos={0 0}, upload=240701]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:40:18.722218529 (250h23m29.31632977s) [upload=240701]: Done uploading segments... Apr 18 02:40:18.722256343 (250h23m29.316367586s) [upload=240701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.661463411s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..XX.X.....XX..X............................................] duration: 1.579284844s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:43:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.579315305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XXXXX.XXX.XXX.....X........XX.......X................X.....] duration: 3.483114533s trace: https://jaeger.dev.storj.io/trace/31005e07d94faebc FAIL (2026-04-18 02:43:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 02:39:18.903757261 (250h22m29.4978685s) [upload=240701]: Got next segment. Inline: false Apr 18 02:39:18.903830365 (250h22m29.497941603s) [upload=240701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:39:18.903837205 (250h22m29.497948444s) [upload=240701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:39:19.014891342 (250h22m29.609002572s) [seg_pos={0 0}, upload=240701]: Begin upload segment... Apr 18 02:39:19.014972185 (250h22m29.609083415s) [seg_pos={0 0}, upload=240701]: Done begin upload segment. Apr 18 02:39:19.015019360 (250h22m29.609130603s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015032108 (250h22m29.609143353s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015104542 (250h22m29.609215781s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015691354 (250h22m29.609802603s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.016299539 (250h22m29.610410788s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017036654 (250h22m29.611147898s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017269327 (250h22m29.611380573s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017746091 (250h22m29.611857335s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018291629 (250h22m29.612402878s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018353743 (250h22m29.612464986s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018721265 (250h22m29.612832515s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018839235 (250h22m29.61295048s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019288928 (250h22m29.613400176s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019746809 (250h22m29.613858054s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020163463 (250h22m29.614274708s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020821300 (250h22m29.614932548s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021249261 (250h22m29.615360503s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021618436 (250h22m29.615729677s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.164125748 (250h22m29.758236994s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:60394->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.200099142 (250h22m29.79421039s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34470->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.580941409 (250h22m30.175052661s) [upload=240701]: Next returned nil segment Apr 18 02:39:19.580961207 (250h22m30.175072451s) [upload=240701]: Waiting for error group managing segments... Apr 18 02:39:19.741177031 (250h22m30.33528829s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.745737016 (250h22m30.339848276s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.789115076 (250h22m30.383226333s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.848298928 (250h22m30.442410169s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.199475147 (250h22m30.793586388s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.319894286 (250h22m30.914005536s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.356429032 (250h22m30.950540277s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.383013757 (250h22m30.977125007s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.384162295 (250h22m30.978273543s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.424713162 (250h22m31.018824409s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.439978786 (250h22m31.034090027s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.476434966 (250h22m31.070546206s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.802472404 (250h22m31.396583644s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:40:18.617604639 (250h23m29.211715882s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:53148->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.617816606 (250h23m29.211927855s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.618072575 (250h23m29.212183825s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:145.79.227.49:4363): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:40:18.618108549 (250h23m29.212219783s) [seg_pos={0 0}, upload=240701]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:40:18.722218529 (250h23m29.31632977s) [upload=240701]: Done uploading segments... Apr 18 02:40:18.722256343 (250h23m29.316367586s) [upload=240701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.483141213s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X..XX.X.....XX..X...........................................] duration: 2.094657281s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:43:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.094750065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [XX..XX.X.....XX..X..........................................] duration: 1.629896341s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:42:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.62992592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.XXXXX.XXX.XXX.....X........XX.......X................X....] duration: 5.708185015s trace: https://jaeger.dev.storj.io/trace/31005e07d94faebc FAIL (2026-04-18 02:42:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 02:39:18.903757261 (250h22m29.4978685s) [upload=240701]: Got next segment. Inline: false Apr 18 02:39:18.903830365 (250h22m29.497941603s) [upload=240701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:39:18.903837205 (250h22m29.497948444s) [upload=240701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:39:19.014891342 (250h22m29.609002572s) [seg_pos={0 0}, upload=240701]: Begin upload segment... Apr 18 02:39:19.014972185 (250h22m29.609083415s) [seg_pos={0 0}, upload=240701]: Done begin upload segment. Apr 18 02:39:19.015019360 (250h22m29.609130603s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015032108 (250h22m29.609143353s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015104542 (250h22m29.609215781s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015691354 (250h22m29.609802603s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.016299539 (250h22m29.610410788s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017036654 (250h22m29.611147898s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017269327 (250h22m29.611380573s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017746091 (250h22m29.611857335s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018291629 (250h22m29.612402878s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018353743 (250h22m29.612464986s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018721265 (250h22m29.612832515s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018839235 (250h22m29.61295048s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019288928 (250h22m29.613400176s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019746809 (250h22m29.613858054s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020163463 (250h22m29.614274708s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020821300 (250h22m29.614932548s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021249261 (250h22m29.615360503s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021618436 (250h22m29.615729677s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.164125748 (250h22m29.758236994s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:60394->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.200099142 (250h22m29.79421039s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34470->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.580941409 (250h22m30.175052661s) [upload=240701]: Next returned nil segment Apr 18 02:39:19.580961207 (250h22m30.175072451s) [upload=240701]: Waiting for error group managing segments... Apr 18 02:39:19.741177031 (250h22m30.33528829s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.745737016 (250h22m30.339848276s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.789115076 (250h22m30.383226333s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.848298928 (250h22m30.442410169s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.199475147 (250h22m30.793586388s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.319894286 (250h22m30.914005536s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.356429032 (250h22m30.950540277s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.383013757 (250h22m30.977125007s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.384162295 (250h22m30.978273543s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.424713162 (250h22m31.018824409s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.439978786 (250h22m31.034090027s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.476434966 (250h22m31.070546206s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.802472404 (250h22m31.396583644s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:40:18.617604639 (250h23m29.211715882s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:53148->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.617816606 (250h23m29.211927855s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.618072575 (250h23m29.212183825s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:145.79.227.49:4363): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:40:18.618108549 (250h23m29.212219783s) [seg_pos={0 0}, upload=240701]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:40:18.722218529 (250h23m29.31632977s) [upload=240701]: Done uploading segments... Apr 18 02:40:18.722256343 (250h23m29.316367586s) [upload=240701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.708225592s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.XX..XX.X.....XX..X.........................................] duration: 1.670719684s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:41:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.670745881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX.XXXXX.XXX.XXX.....X........XX.......X................X...] duration: 3.537828641s trace: https://jaeger.dev.storj.io/trace/31005e07d94faebc FAIL (2026-04-18 02:41:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 02:39:18.903757261 (250h22m29.4978685s) [upload=240701]: Got next segment. Inline: false Apr 18 02:39:18.903830365 (250h22m29.497941603s) [upload=240701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:39:18.903837205 (250h22m29.497948444s) [upload=240701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:39:19.014891342 (250h22m29.609002572s) [seg_pos={0 0}, upload=240701]: Begin upload segment... Apr 18 02:39:19.014972185 (250h22m29.609083415s) [seg_pos={0 0}, upload=240701]: Done begin upload segment. Apr 18 02:39:19.015019360 (250h22m29.609130603s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015032108 (250h22m29.609143353s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015104542 (250h22m29.609215781s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015691354 (250h22m29.609802603s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.016299539 (250h22m29.610410788s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017036654 (250h22m29.611147898s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017269327 (250h22m29.611380573s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017746091 (250h22m29.611857335s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018291629 (250h22m29.612402878s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018353743 (250h22m29.612464986s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018721265 (250h22m29.612832515s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018839235 (250h22m29.61295048s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019288928 (250h22m29.613400176s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019746809 (250h22m29.613858054s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020163463 (250h22m29.614274708s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020821300 (250h22m29.614932548s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021249261 (250h22m29.615360503s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021618436 (250h22m29.615729677s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.164125748 (250h22m29.758236994s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:60394->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.200099142 (250h22m29.79421039s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34470->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.580941409 (250h22m30.175052661s) [upload=240701]: Next returned nil segment Apr 18 02:39:19.580961207 (250h22m30.175072451s) [upload=240701]: Waiting for error group managing segments... Apr 18 02:39:19.741177031 (250h22m30.33528829s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.745737016 (250h22m30.339848276s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.789115076 (250h22m30.383226333s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.848298928 (250h22m30.442410169s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.199475147 (250h22m30.793586388s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.319894286 (250h22m30.914005536s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.356429032 (250h22m30.950540277s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.383013757 (250h22m30.977125007s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.384162295 (250h22m30.978273543s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.424713162 (250h22m31.018824409s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.439978786 (250h22m31.034090027s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.476434966 (250h22m31.070546206s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.802472404 (250h22m31.396583644s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:40:18.617604639 (250h23m29.211715882s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:53148->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.617816606 (250h23m29.211927855s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.618072575 (250h23m29.212183825s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:145.79.227.49:4363): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:40:18.618108549 (250h23m29.212219783s) [seg_pos={0 0}, upload=240701]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:40:18.722218529 (250h23m29.31632977s) [upload=240701]: Done uploading segments... Apr 18 02:40:18.722256343 (250h23m29.316367586s) [upload=240701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.537853856s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X.XX..XX.X.....XX..X........................................] duration: 2.177890384s trace: https://jaeger.dev.storj.io/trace/15bcaf84944be57a FAIL (2026-04-18 02:40:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.177924816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXX.XXXXX.XXX.XXX.....X........XX.......X................X..] duration: 3.711085666s trace: https://jaeger.dev.storj.io/trace/31005e07d94faebc FAIL (2026-04-18 02:40:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 02:39:18.903757261 (250h22m29.4978685s) [upload=240701]: Got next segment. Inline: false Apr 18 02:39:18.903830365 (250h22m29.497941603s) [upload=240701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 02:39:18.903837205 (250h22m29.497948444s) [upload=240701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 02:39:19.014891342 (250h22m29.609002572s) [seg_pos={0 0}, upload=240701]: Begin upload segment... Apr 18 02:39:19.014972185 (250h22m29.609083415s) [seg_pos={0 0}, upload=240701]: Done begin upload segment. Apr 18 02:39:19.015019360 (250h22m29.609130603s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015032108 (250h22m29.609143353s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015104542 (250h22m29.609215781s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.015691354 (250h22m29.609802603s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.016299539 (250h22m29.610410788s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017036654 (250h22m29.611147898s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017269327 (250h22m29.611380573s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.017746091 (250h22m29.611857335s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018291629 (250h22m29.612402878s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018353743 (250h22m29.612464986s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018721265 (250h22m29.612832515s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.018839235 (250h22m29.61295048s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019288928 (250h22m29.613400176s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.019746809 (250h22m29.613858054s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020163463 (250h22m29.614274708s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.020821300 (250h22m29.614932548s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021249261 (250h22m29.615360503s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.021618436 (250h22m29.615729677s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Uploading piece... Apr 18 02:39:19.164125748 (250h22m29.758236994s) [noise=true , address=167.17.72.69:4725 , piece_id=IGC5PDAEABCPFEYDFSQEWX5I2S5CZTXZ4C2437VR6OGUYGCDHNYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1FgFAvkTus7ri5TcLGjMv2LVMa5e4fYsj898RSyjdn1EGBEs51, address:167.17.72.69:4725): protocol: write tcp 139.59.144.22:60394->167.17.72.69:4725: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.200099142 (250h22m29.79421039s) [noise=true , address=193.149.164.143:4696 , piece_id=V4PEZEHTEIH4WPFPIJF4JBLGUWQMDUWUPO4ZB6VF2FVNAPOV4BNQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: write tcp 139.59.144.22:34470->193.149.164.143:4696: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:39:19.580941409 (250h22m30.175052661s) [upload=240701]: Next returned nil segment Apr 18 02:39:19.580961207 (250h22m30.175072451s) [upload=240701]: Waiting for error group managing segments... Apr 18 02:39:19.741177031 (250h22m30.33528829s) [noise=true , address=45.91.116.13:4167 , piece_id=HRAPLPJ2TRAXWUDGTAINGPVMTLPASFFHSUA7UORCAWHOEJT3PATQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.745737016 (250h22m30.339848276s) [noise=true , address=145.79.227.32:4864 , piece_id=2K22I4OTC4SMKA2AZIOB4ZRFZ3NHQZB25TRTC53AHT5DRGWBJGYQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.789115076 (250h22m30.383226333s) [noise=true , address=45.91.116.8:4063 , piece_id=AQMVSSGTR2WR4UPHD3QKINIUOXRECQ2LRHHO2A2WTQV3PKQZA26Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:19.848298928 (250h22m30.442410169s) [noise=true , address=145.79.227.34:4929 , piece_id=HNZOJTF3P3YX3D722PS2ERCBTZZYMS6KCX7S4WNEZDVMYD7FXYIA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.199475147 (250h22m30.793586388s) [noise=true , address=145.79.227.11:4747 , piece_id=GMJR4W6BC3ARXHQHEL5JAZK35D3FI3BRXTGFGTS5TGKLN3OI3BQA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.319894286 (250h22m30.914005536s) [noise=true , address=167.17.72.70:4775 , piece_id=JJ2D3SP4JNBOYARQGSJMTLFPUAXDGIBNJG5GFSW6KWL4ZN3B37OQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.356429032 (250h22m30.950540277s) [noise=true , address=64.49.13.139:4474 , piece_id=GRAGAPLOC6KZ545KIMQ4E4FEWFOE2QETB7DTRH3NBCLGXQO2O4PQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.383013757 (250h22m30.977125007s) [noise=true , address=79.127.205.234:4195 , piece_id=RRNZ6DWBP4XFAX5GBIQGCORR6BZFLXJPG6EZ67SRUGVRUQVKVUWA, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.384162295 (250h22m30.978273543s) [noise=true , address=79.127.205.233:4170 , piece_id=O5S24QNL67HQ5KKRIYWMFSHHCSME22CMTRQTWKCXXBNWFMVJRMSQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.424713162 (250h22m31.018824409s) [noise=true , address=193.149.164.71:4318 , piece_id=FQ5VMRUBNVROAP2BCN65KL2WFSIVDVSSVBRCFAS2RODJPMRPI7YQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.439978786 (250h22m31.034090027s) [noise=true , address=79.127.205.235:4268 , piece_id=YF7P3UEIGMVKZCXNBSEAXJ3A4OB7Q733AYFM7RZA45AO2GBIPNFQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.476434966 (250h22m31.070546206s) [noise=true , address=79.127.205.237:4213 , piece_id=H4ELELGY63BIXNJ64CB4W2EF3ORGIMM5DIPFXNGJYQMMBYJB6VIQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:39:20.802472404 (250h22m31.396583644s) [noise=true , address=145.79.227.44:4206 , piece_id=DKIG2SSMJZF6OKSZRTXRTSCSY75AP4DIN2HF56BZ6MPK6YWA445Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: <nil> Apr 18 02:40:18.617604639 (250h23m29.211715882s) [noise=true , address=167.17.72.71:4809 , piece_id=BDBML3JCEKMB3G4K7I4RR4YG3MLAAKHHA64KGPNN5W6X6ILBXU7Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:53148->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.617816606 (250h23m29.211927855s) [noise=true , address=145.79.227.38:4010 , piece_id=ECCT3YE7PAOXIECGZVKZDZTDXKJASVK3TFXNSRJ44K5F3O5D53QQ, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 02:40:18.618072575 (250h23m29.212183825s) [noise=true , address=145.79.227.49:4363 , piece_id=ZHYOZAUYXGT4WLMURJVEBCAK2ZZMO7R7OUZAERL4IC3FKTMQL65Q, seg_pos={0 0}, upload=240701]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:145.79.227.49:4363): protocol: context deadline exceeded; context deadline exceeded Apr 18 02:40:18.618108549 (250h23m29.212219783s) [seg_pos={0 0}, upload=240701]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 02:40:18.722218529 (250h23m29.31632977s) [upload=240701]: Done uploading segments... Apr 18 02:40:18.722256343 (250h23m29.316367586s) [upload=240701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.711131583s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)