Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X.......X..X.] duration: 22.754009233s FAIL (2025-07-26 22:25:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.754062958s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...............X.X..X..XXX....X.X.X...............X.........] duration: 2.354884613s FAIL (2025-07-26 22:25:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.354918211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.110897854s FAIL (2025-07-26 22:25:29 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 1.110941513s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [............XX.X..................X.X.......................] duration: 2.237652466s FAIL (2025-07-26 22:25:13 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.237682156s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [................X.X..X..XXX....X.X.X...............X........] duration: 2.434172792s FAIL (2025-07-26 22:25:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.434212501s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................................................X.......X..X] duration: 1m0.000271848s FAIL (2025-07-26 22:25:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000290628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.049769872s FAIL (2025-07-26 22:25:09 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 1.04980248s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...................................X.............XX...XXXXX.] duration: 56.701922559s FAIL (2025-07-26 22:25:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 26 22:23:06.174390512 (58h47m37.364689098s) [noise=true , address=79.127.219.47:4061 , piece_id=IWXJRMLJOH5JXGWV5GBDA4IVLBMOREEARCFJZLSH5YVZMOSJIANQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.174423094 (58h47m37.364721679s) [noise=true , address=104.36.82.55:4228 , piece_id=4JOTEBQI47YHOAGIQZF67MUWSG63FXV5IJ6UDG7WZWPRWKPY3JYQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.174758957 (58h47m37.365057539s) [noise=true , address=64.49.13.140:4508 , piece_id=2PAY47CCLUKP4VW4SHAJJK3SZJOL3YM7XHBVNFZNCR365FKDYRJA, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.175032712 (58h47m37.365331295s) [noise=true , address=104.36.82.42:4811 , piece_id=AW5NZS3X7MM42NY2ZBXRMETWWZSCSZ2BGR2A6XSS2SPJAA67BVWQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.175277594 (58h47m37.365576173s) [noise=true , address=77.90.31.17:4796 , piece_id=X2LBVRLXVZ2EYWVDF6SOLSIYCDO75VSGOR6DIQVOSKDEO4NVJWKA, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.175628850 (58h47m37.36592744s) [noise=true , address=104.36.82.2:4752 , piece_id=J2HLIH5WC7BOOZ55634GOWUDNWSBF45FR3KIRZUTQQLCNWR5S5QQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.176020698 (58h47m37.366319283s) [noise=true , address=104.36.82.51:4080 , piece_id=I3YUZLE2ZTXYWEA2FIKV7P25ACLHCILF5W5QSFZHIYLRLOY5Y3AQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.176451530 (58h47m37.366750112s) [noise=true , address=77.90.31.18:4821 , piece_id=U7MRS72NUGGO6KYZUDNXC2OYYST2XI2UBSJSOMZ3SLHWBTVD3V7Q, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.177368720 (58h47m37.367667308s) [noise=true , address=79.127.205.235:4680 , piece_id=AYAREFLE2NLS6YFVS6S6AZIKHPPYOHJCOMU4UQPEHMMXGOW45X4A, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.178176393 (58h47m37.368474982s) [noise=true , address=104.36.82.56:4230 , piece_id=AW4OC6WAS7SCDJG2JFFQ3APX2AN6SVBP6PTF2CQJ22HCQJE6CTMA, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.179205747 (58h47m37.369504334s) [noise=true , address=156.146.43.225:4434 , piece_id=TNAGDYRQSWEQM43XETGYXPFAURKQMEDI3NOQENI464E2JFINRXLA, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.180261973 (58h47m37.370560561s) [noise=true , address=104.36.82.35:4615 , piece_id=GXQDKEDNZQFNBNENQR4ZMRTGMCZQNSU7QSKWYKBUDMGLQC2BIJIA, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.181130708 (58h47m37.371429296s) [noise=true , address=79.127.220.101:4048 , piece_id=ON2XICT4N6IUC2JYBMUFRKYHC2XWQNVRZRFWMCEIMGDLPCIC72GQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.181482842 (58h47m37.371781427s) [noise=true , address=79.127.219.51:4210 , piece_id=PQ3RQ4KT7EAIJO2WN63VF524IVV6AATWTNORPZL2ZWMLZPK7C4MA, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.182041266 (58h47m37.372339854s) [noise=true , address=104.36.82.40:4952 , piece_id=DGT4GI26QITV74SK674QXW5FIQ54CPQASXAPBCQJ64DOU43U6TOQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.183389887 (58h47m37.373688469s) [noise=true , address=45.91.116.14:4222 , piece_id=6R2YUQ53W7IET3QIEZHTS3F2KFHB3WKZ7RT52ISKN33JOSJHW2QQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.756347432 (58h47m37.946646013s) [upload=56385]: Next returned nil segment Jul 26 22:23:06.756372676 (58h47m37.946671263s) [upload=56385]: Waiting for error group managing segments... Jul 26 22:23:07.011600519 (58h47m38.201899093s) [noise=true , address=77.90.31.18:4821 , piece_id=U7MRS72NUGGO6KYZUDNXC2OYYST2XI2UBSJSOMZ3SLHWBTVD3V7Q, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:07.368674695 (58h47m38.558973287s) [noise=true , address=104.36.82.55:4228 , piece_id=4JOTEBQI47YHOAGIQZF67MUWSG63FXV5IJ6UDG7WZWPRWKPY3JYQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:07.478313294 (58h47m38.668611884s) [noise=true , address=79.127.205.235:4680 , piece_id=AYAREFLE2NLS6YFVS6S6AZIKHPPYOHJCOMU4UQPEHMMXGOW45X4A, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:07.727776440 (58h47m38.918075024s) [noise=true , address=156.146.43.225:4434 , piece_id=TNAGDYRQSWEQM43XETGYXPFAURKQMEDI3NOQENI464E2JFINRXLA, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:08.320977576 (58h47m39.511276157s) [noise=true , address=79.127.219.51:4210 , piece_id=PQ3RQ4KT7EAIJO2WN63VF524IVV6AATWTNORPZL2ZWMLZPK7C4MA, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:09.053527336 (58h47m40.243825916s) [noise=true , address=79.127.219.47:4061 , piece_id=IWXJRMLJOH5JXGWV5GBDA4IVLBMOREEARCFJZLSH5YVZMOSJIANQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:09.301537747 (58h47m40.49183632s) [noise=true , address=79.127.220.101:4048 , piece_id=ON2XICT4N6IUC2JYBMUFRKYHC2XWQNVRZRFWMCEIMGDLPCIC72GQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:14.675067816 (58h47m45.865366386s) [noise=true , address=45.91.116.14:4222 , piece_id=6R2YUQ53W7IET3QIEZHTS3F2KFHB3WKZ7RT52ISKN33JOSJHW2QQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:16.502185120 (58h47m47.692483704s) [noise=true , address=64.49.13.140:4508 , piece_id=2PAY47CCLUKP4VW4SHAJJK3SZJOL3YM7XHBVNFZNCR365FKDYRJA, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:24.645395486 (58h47m55.835694066s) [noise=true , address=104.36.82.56:4230 , piece_id=AW4OC6WAS7SCDJG2JFFQ3APX2AN6SVBP6PTF2CQJ22HCQJE6CTMA, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:29.682480506 (58h48m0.872779071s) [noise=true , address=104.36.82.2:4752 , piece_id=J2HLIH5WC7BOOZ55634GOWUDNWSBF45FR3KIRZUTQQLCNWR5S5QQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:24:05.707060016 (58h48m36.897358596s) [noise=true , address=104.36.82.40:4952 , piece_id=DGT4GI26QITV74SK674QXW5FIQ54CPQASXAPBCQJ64DOU43U6TOQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:1QucqyXk8Zx4q4uK6nMULHHRjZCRfQDoUREoHZNEjsQwVQt7gK, address:104.36.82.40:4952): protocol: write tcp 139.59.144.22:54526->104.36.82.40:4952: use of closed network connection; write tcp 139.59.144.22:54526->104.36.82.40:4952: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54526->104.36.82.40:4952: use of closed network connection Jul 26 22:24:05.707173513 (58h48m36.897472093s) [noise=true , address=104.36.82.35:4615 , piece_id=GXQDKEDNZQFNBNENQR4ZMRTGMCZQNSU7QSKWYKBUDMGLQC2BIJIA, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: write tcp 139.59.144.22:45010->104.36.82.35:4615: use of closed network connection; write tcp 139.59.144.22:45010->104.36.82.35:4615: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45010->104.36.82.35:4615: use of closed network connection Jul 26 22:24:05.707257019 (58h48m36.897555602s) [noise=true , address=104.36.82.42:4811 , piece_id=AW5NZS3X7MM42NY2ZBXRMETWWZSCSZ2BGR2A6XSS2SPJAA67BVWQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: write tcp 139.59.144.22:44370->104.36.82.42:4811: use of closed network connection; write tcp 139.59.144.22:44370->104.36.82.42:4811: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44370->104.36.82.42:4811: use of closed network connection Jul 26 22:24:05.707484484 (58h48m36.897783075s) [noise=true , address=77.90.31.17:4796 , piece_id=X2LBVRLXVZ2EYWVDF6SOLSIYCDO75VSGOR6DIQVOSKDEO4NVJWKA, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:24:05.707794193 (58h48m36.898092772s) [noise=true , address=77.90.31.12:4612 , piece_id=SG23TXE2NVIJ2ULWCVQ3AZAUZNP4BDZQZG3KZBXOK5YN7XM56O2Q, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W, address:77.90.31.12:4612): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4612: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:05.707950008 (58h48m36.898248587s) [noise=true , address=104.36.82.40:4959 , piece_id=3C33AV2KQVATBSXOIPS23OHKJA2AUTF5O6SCEWVSOVFQ24IGC6UQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:1NCN7sTw12KVDpD8zLjUxQki2MdMuDJyEnPSirG8ni5L2xi7aa, address:104.36.82.40:4959): protocol: write tcp 139.59.144.22:60972->104.36.82.40:4959: use of closed network connection; write tcp 139.59.144.22:60972->104.36.82.40:4959: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60972->104.36.82.40:4959: use of closed network connection Jul 26 22:24:05.708091664 (58h48m36.898390244s) [noise=true , address=104.36.82.51:4080 , piece_id=I3YUZLE2ZTXYWEA2FIKV7P25ACLHCILF5W5QSFZHIYLRLOY5Y3AQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:1WXMdaUQnFnXfayz6CkmsK1wuLhWSpXjQQt2kZyVTVnv3qL44C, address:104.36.82.51:4080): protocol: expected piece hash; context deadline exceeded; context deadline exceeded; EOF Jul 26 22:24:05.708296653 (58h48m36.898595301s) [seg_pos={0 0}, upload=56385]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:24:05.820653385 (58h48m37.010951965s) [upload=56385]: Done uploading segments... Jul 26 22:24:05.820682971 (58h48m37.010981551s) [upload=56385]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 56.701955135s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.685566779s FAIL (2025-07-26 22:24:47 UTC) orbiter warning: context deadline exceeded Duration: 2.685599322s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............XX.X..................X.X......................] duration: 2.826536462s FAIL (2025-07-26 22:24:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.826565302s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.................X.X..X..XXX....X.X.X...............X.......] duration: 2.369856761s FAIL (2025-07-26 22:24:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.369885747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............XX.X..................X.X.....................] duration: 2.247738209s FAIL (2025-07-26 22:24:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.247769267s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....................................X.............XX...XXXXX] duration: 1m0.115313151s FAIL (2025-07-26 22:24:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 26 22:23:06.174390512 (58h47m37.364689098s) [noise=true , address=79.127.219.47:4061 , piece_id=IWXJRMLJOH5JXGWV5GBDA4IVLBMOREEARCFJZLSH5YVZMOSJIANQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.174423094 (58h47m37.364721679s) [noise=true , address=104.36.82.55:4228 , piece_id=4JOTEBQI47YHOAGIQZF67MUWSG63FXV5IJ6UDG7WZWPRWKPY3JYQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.174758957 (58h47m37.365057539s) [noise=true , address=64.49.13.140:4508 , piece_id=2PAY47CCLUKP4VW4SHAJJK3SZJOL3YM7XHBVNFZNCR365FKDYRJA, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.175032712 (58h47m37.365331295s) [noise=true , address=104.36.82.42:4811 , piece_id=AW5NZS3X7MM42NY2ZBXRMETWWZSCSZ2BGR2A6XSS2SPJAA67BVWQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.175277594 (58h47m37.365576173s) [noise=true , address=77.90.31.17:4796 , piece_id=X2LBVRLXVZ2EYWVDF6SOLSIYCDO75VSGOR6DIQVOSKDEO4NVJWKA, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.175628850 (58h47m37.36592744s) [noise=true , address=104.36.82.2:4752 , piece_id=J2HLIH5WC7BOOZ55634GOWUDNWSBF45FR3KIRZUTQQLCNWR5S5QQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.176020698 (58h47m37.366319283s) [noise=true , address=104.36.82.51:4080 , piece_id=I3YUZLE2ZTXYWEA2FIKV7P25ACLHCILF5W5QSFZHIYLRLOY5Y3AQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.176451530 (58h47m37.366750112s) [noise=true , address=77.90.31.18:4821 , piece_id=U7MRS72NUGGO6KYZUDNXC2OYYST2XI2UBSJSOMZ3SLHWBTVD3V7Q, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.177368720 (58h47m37.367667308s) [noise=true , address=79.127.205.235:4680 , piece_id=AYAREFLE2NLS6YFVS6S6AZIKHPPYOHJCOMU4UQPEHMMXGOW45X4A, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.178176393 (58h47m37.368474982s) [noise=true , address=104.36.82.56:4230 , piece_id=AW4OC6WAS7SCDJG2JFFQ3APX2AN6SVBP6PTF2CQJ22HCQJE6CTMA, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.179205747 (58h47m37.369504334s) [noise=true , address=156.146.43.225:4434 , piece_id=TNAGDYRQSWEQM43XETGYXPFAURKQMEDI3NOQENI464E2JFINRXLA, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.180261973 (58h47m37.370560561s) [noise=true , address=104.36.82.35:4615 , piece_id=GXQDKEDNZQFNBNENQR4ZMRTGMCZQNSU7QSKWYKBUDMGLQC2BIJIA, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.181130708 (58h47m37.371429296s) [noise=true , address=79.127.220.101:4048 , piece_id=ON2XICT4N6IUC2JYBMUFRKYHC2XWQNVRZRFWMCEIMGDLPCIC72GQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.181482842 (58h47m37.371781427s) [noise=true , address=79.127.219.51:4210 , piece_id=PQ3RQ4KT7EAIJO2WN63VF524IVV6AATWTNORPZL2ZWMLZPK7C4MA, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.182041266 (58h47m37.372339854s) [noise=true , address=104.36.82.40:4952 , piece_id=DGT4GI26QITV74SK674QXW5FIQ54CPQASXAPBCQJ64DOU43U6TOQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.183389887 (58h47m37.373688469s) [noise=true , address=45.91.116.14:4222 , piece_id=6R2YUQ53W7IET3QIEZHTS3F2KFHB3WKZ7RT52ISKN33JOSJHW2QQ, seg_pos={0 0}, upload=56385]: Uploading piece... Jul 26 22:23:06.756347432 (58h47m37.946646013s) [upload=56385]: Next returned nil segment Jul 26 22:23:06.756372676 (58h47m37.946671263s) [upload=56385]: Waiting for error group managing segments... Jul 26 22:23:07.011600519 (58h47m38.201899093s) [noise=true , address=77.90.31.18:4821 , piece_id=U7MRS72NUGGO6KYZUDNXC2OYYST2XI2UBSJSOMZ3SLHWBTVD3V7Q, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:07.368674695 (58h47m38.558973287s) [noise=true , address=104.36.82.55:4228 , piece_id=4JOTEBQI47YHOAGIQZF67MUWSG63FXV5IJ6UDG7WZWPRWKPY3JYQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:07.478313294 (58h47m38.668611884s) [noise=true , address=79.127.205.235:4680 , piece_id=AYAREFLE2NLS6YFVS6S6AZIKHPPYOHJCOMU4UQPEHMMXGOW45X4A, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:07.727776440 (58h47m38.918075024s) [noise=true , address=156.146.43.225:4434 , piece_id=TNAGDYRQSWEQM43XETGYXPFAURKQMEDI3NOQENI464E2JFINRXLA, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:08.320977576 (58h47m39.511276157s) [noise=true , address=79.127.219.51:4210 , piece_id=PQ3RQ4KT7EAIJO2WN63VF524IVV6AATWTNORPZL2ZWMLZPK7C4MA, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:09.053527336 (58h47m40.243825916s) [noise=true , address=79.127.219.47:4061 , piece_id=IWXJRMLJOH5JXGWV5GBDA4IVLBMOREEARCFJZLSH5YVZMOSJIANQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:09.301537747 (58h47m40.49183632s) [noise=true , address=79.127.220.101:4048 , piece_id=ON2XICT4N6IUC2JYBMUFRKYHC2XWQNVRZRFWMCEIMGDLPCIC72GQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:14.675067816 (58h47m45.865366386s) [noise=true , address=45.91.116.14:4222 , piece_id=6R2YUQ53W7IET3QIEZHTS3F2KFHB3WKZ7RT52ISKN33JOSJHW2QQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:16.502185120 (58h47m47.692483704s) [noise=true , address=64.49.13.140:4508 , piece_id=2PAY47CCLUKP4VW4SHAJJK3SZJOL3YM7XHBVNFZNCR365FKDYRJA, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:24.645395486 (58h47m55.835694066s) [noise=true , address=104.36.82.56:4230 , piece_id=AW4OC6WAS7SCDJG2JFFQ3APX2AN6SVBP6PTF2CQJ22HCQJE6CTMA, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:23:29.682480506 (58h48m0.872779071s) [noise=true , address=104.36.82.2:4752 , piece_id=J2HLIH5WC7BOOZ55634GOWUDNWSBF45FR3KIRZUTQQLCNWR5S5QQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: <nil> Jul 26 22:24:05.707060016 (58h48m36.897358596s) [noise=true , address=104.36.82.40:4952 , piece_id=DGT4GI26QITV74SK674QXW5FIQ54CPQASXAPBCQJ64DOU43U6TOQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:1QucqyXk8Zx4q4uK6nMULHHRjZCRfQDoUREoHZNEjsQwVQt7gK, address:104.36.82.40:4952): protocol: write tcp 139.59.144.22:54526->104.36.82.40:4952: use of closed network connection; write tcp 139.59.144.22:54526->104.36.82.40:4952: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54526->104.36.82.40:4952: use of closed network connection Jul 26 22:24:05.707173513 (58h48m36.897472093s) [noise=true , address=104.36.82.35:4615 , piece_id=GXQDKEDNZQFNBNENQR4ZMRTGMCZQNSU7QSKWYKBUDMGLQC2BIJIA, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: write tcp 139.59.144.22:45010->104.36.82.35:4615: use of closed network connection; write tcp 139.59.144.22:45010->104.36.82.35:4615: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45010->104.36.82.35:4615: use of closed network connection Jul 26 22:24:05.707257019 (58h48m36.897555602s) [noise=true , address=104.36.82.42:4811 , piece_id=AW5NZS3X7MM42NY2ZBXRMETWWZSCSZ2BGR2A6XSS2SPJAA67BVWQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: write tcp 139.59.144.22:44370->104.36.82.42:4811: use of closed network connection; write tcp 139.59.144.22:44370->104.36.82.42:4811: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44370->104.36.82.42:4811: use of closed network connection Jul 26 22:24:05.707484484 (58h48m36.897783075s) [noise=true , address=77.90.31.17:4796 , piece_id=X2LBVRLXVZ2EYWVDF6SOLSIYCDO75VSGOR6DIQVOSKDEO4NVJWKA, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:24:05.707794193 (58h48m36.898092772s) [noise=true , address=77.90.31.12:4612 , piece_id=SG23TXE2NVIJ2ULWCVQ3AZAUZNP4BDZQZG3KZBXOK5YN7XM56O2Q, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W, address:77.90.31.12:4612): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4612: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:05.707950008 (58h48m36.898248587s) [noise=true , address=104.36.82.40:4959 , piece_id=3C33AV2KQVATBSXOIPS23OHKJA2AUTF5O6SCEWVSOVFQ24IGC6UQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:1NCN7sTw12KVDpD8zLjUxQki2MdMuDJyEnPSirG8ni5L2xi7aa, address:104.36.82.40:4959): protocol: write tcp 139.59.144.22:60972->104.36.82.40:4959: use of closed network connection; write tcp 139.59.144.22:60972->104.36.82.40:4959: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60972->104.36.82.40:4959: use of closed network connection Jul 26 22:24:05.708091664 (58h48m36.898390244s) [noise=true , address=104.36.82.51:4080 , piece_id=I3YUZLE2ZTXYWEA2FIKV7P25ACLHCILF5W5QSFZHIYLRLOY5Y3AQ, seg_pos={0 0}, upload=56385]: Done uploading piece. err: ecclient: upload failed (node:1WXMdaUQnFnXfayz6CkmsK1wuLhWSpXjQQt2kZyVTVnv3qL44C, address:104.36.82.51:4080): protocol: expected piece hash; context deadline exceeded; context deadline exceeded; EOF Jul 26 22:24:05.708296653 (58h48m36.898595301s) [seg_pos={0 0}, upload=56385]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:24:05.820653385 (58h48m37.010951965s) [upload=56385]: Done uploading segments... Jul 26 22:24:05.820682971 (58h48m37.010981551s) [upload=56385]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.115347206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............XX.X..................X.X....................] duration: 2.873251554s FAIL (2025-07-26 22:24:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.873281783s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 854.708501ms FAIL (2025-07-26 22:23:58 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 854.742488ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [................XX.X..................X.X...................] duration: 2.3352056s FAIL (2025-07-26 22:23:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.335238408s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.665916872s FAIL (2025-07-26 22:23:36 UTC) orbiter warning: context deadline exceeded Duration: 2.665949342s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................X.X..X..XXX....X.X.X...............X......] duration: 2.347013378s FAIL (2025-07-26 22:23:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.347110442s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.......X..] duration: 5.251257037s FAIL (2025-07-26 22:23:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.251284514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 852.366002ms FAIL (2025-07-26 22:23:18 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 852.397995ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....................................X.............XX...XXXX] duration: 1m0.115010969s FAIL (2025-07-26 22:23:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== [upload=56369]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:22:05.911269880 (58h46m37.101568449s) [seg_pos={0 0}, upload=56369]: Begin upload segment... Jul 26 22:22:05.911330901 (58h46m37.101629468s) [seg_pos={0 0}, upload=56369]: Done begin upload segment. Jul 26 22:22:05.911359493 (58h46m37.101658075s) [noise=true , address=104.36.82.13:4193 , piece_id=DHTUE25QWQVB7KFJHWQNBXYNCGFPN5B4WLH7OTQYOMIVHLDMKCBQ, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.911606398 (58h46m37.101904969s) [noise=true , address=104.36.82.18:4393 , piece_id=FWD77PNVTHRCVR3MONVKFDMGUQWNYFMVOYTAHQINTA6MWSYVTGPQ, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.911942691 (58h46m37.102241264s) [noise=true , address=104.36.82.41:4762 , piece_id=JAKPPT76JPKAAQVETWT5Y6DKSKH6YDO6RH6BM7IK5N2GYVJALM4Q, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.915299753 (58h46m37.105598335s) [noise=true , address=104.36.82.5:4851 , piece_id=G274D6QIAY7LUO7L63C2ZRAB4TUB3MHK5GSMVCWM7DWMWYFBVO6A, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.915690737 (58h46m37.105989311s) [noise=true , address=79.127.205.233:4647 , piece_id=3P3SYNDTCP4DZAMAMYQADZ2HSI3OGQ7U4BNPG5ADQZ3ZFKHHCOKA, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.916577589 (58h46m37.106876161s) [noise=true , address=104.36.82.39:4757 , piece_id=CPSCZA4I2LQI2MXJPJTVHFBTXKL7QRIXVTHNVL22EB3ML55ZFNWA, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.917203134 (58h46m37.107501708s) [noise=true , address=104.36.82.35:4592 , piece_id=7XRGNLGYRQ6TXZW72ROZUS2RYLVZIACFYIRQBMPCS7HEF5SNP4GA, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.917853043 (58h46m37.108151614s) [noise=true , address=104.36.82.12:4180 , piece_id=6X4GG65QCJKDEHONU6AIA6MPKGI6GJRNUQ22SWQIIMU7JDS362WA, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.919682837 (58h46m37.10998141s) [noise=true , address=104.36.82.34:4561 , piece_id=HIPZEOWJERVJAIGWGRP5EYFA4OFZVFQV2DEZV6M4FYRLFDA2BSQQ, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.921651265 (58h46m37.111949847s) [noise=true , address=104.36.82.56:4231 , piece_id=YB2DGU2NSWER7BVROEFKJCWIPMGEGZARGCCTGOCEOZRGEGBRMJTA, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.923669019 (58h46m37.113967591s) [noise=true , address=45.91.116.10:4100 , piece_id=Y5D3VD4ASPDNHLUOFX4ZRZSBKXUTONCLNP6I42I2SVAQ2W4YHVVA, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.925747192 (58h46m37.116045765s) [noise=true , address=79.127.201.215:4354 , piece_id=DQLAS4XZR645YFIQASJBE7X6HKC2XVDZV5K5US3Q76555AR3CJ5A, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.926393417 (58h46m37.11669199s) [noise=true , address=77.90.31.13:4651 , piece_id=P5VJKPWCN37UCEXBRTQZFBKHJJEGJQ7C6L3HXP7OZ4USPNIIYVHA, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.927060083 (58h46m37.117358662s) [noise=true , address=104.36.82.38:4722 , piece_id=WPL7ZECB5VUOP5XROZBAL5BBOCSDNOAWMGUAKWLX23CERGOP6ZBA, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.929154621 (58h46m37.119453193s) [noise=true , address=79.127.205.245:4238 , piece_id=WBGJUAGWXBKRHTXTL7LOGKRHPXURAUDXG3BLXPUBAMCCEY67GHKA, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.929654392 (58h46m37.119952962s) [noise=true , address=77.90.31.19:4871 , piece_id=FUNU7WPLY5XXXO5FCRU4YYG4R4W3RRYV4S5MKDLKBJQDUC57SAAQ, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.930798671 (58h46m37.121097243s) [noise=true , address=64.49.13.138:4483 , piece_id=KG6XOJKTRW43CS2O6JGC7JZZCBKLIPFBESYUW5W7ZTDZ6FDMG6SQ, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:05.931395329 (58h46m37.121693902s) [noise=true , address=104.36.82.55:4228 , piece_id=FHILU7PQ7EAXU2J662YQKE3PGOIHNL77T7BD43TMXFNGJOA2ORCQ, seg_pos={0 0}, upload=56369]: Uploading piece... Jul 26 22:22:06.495736259 (58h46m37.686034841s) [upload=56369]: Next returned nil segment Jul 26 22:22:06.495776539 (58h46m37.686075125s) [upload=56369]: Waiting for error group managing segments... Jul 26 22:22:06.634952747 (58h46m37.825251325s) [noise=true , address=45.91.116.10:4100 , piece_id=Y5D3VD4ASPDNHLUOFX4ZRZSBKXUTONCLNP6I42I2SVAQ2W4YHVVA, seg_pos={0 0}, upload=56369]: Done uploading piece. err: <nil> Jul 26 22:22:06.649983642 (58h46m37.840282212s) [noise=true , address=104.36.82.56:4231 , piece_id=YB2DGU2NSWER7BVROEFKJCWIPMGEGZARGCCTGOCEOZRGEGBRMJTA, seg_pos={0 0}, upload=56369]: Done uploading piece. err: <nil> Jul 26 22:22:06.806349328 (58h46m37.996647909s) [noise=true , address=104.36.82.55:4228 , piece_id=FHILU7PQ7EAXU2J662YQKE3PGOIHNL77T7BD43TMXFNGJOA2ORCQ, seg_pos={0 0}, upload=56369]: Done uploading piece. err: <nil> Jul 26 22:22:07.213302066 (58h46m38.403600651s) [noise=true , address=79.127.205.245:4238 , piece_id=WBGJUAGWXBKRHTXTL7LOGKRHPXURAUDXG3BLXPUBAMCCEY67GHKA, seg_pos={0 0}, upload=56369]: Done uploading piece. err: <nil> Jul 26 22:22:07.217053565 (58h46m38.407352131s) [noise=true , address=79.127.205.233:4647 , piece_id=3P3SYNDTCP4DZAMAMYQADZ2HSI3OGQ7U4BNPG5ADQZ3ZFKHHCOKA, seg_pos={0 0}, upload=56369]: Done uploading piece. err: <nil> Jul 26 22:22:07.509287884 (58h46m38.699586451s) [noise=true , address=77.90.31.19:4871 , piece_id=FUNU7WPLY5XXXO5FCRU4YYG4R4W3RRYV4S5MKDLKBJQDUC57SAAQ, seg_pos={0 0}, upload=56369]: Done uploading piece. err: <nil> Jul 26 22:22:11.205816936 (58h46m42.396115516s) [noise=true , address=64.49.13.138:4483 , piece_id=KG6XOJKTRW43CS2O6JGC7JZZCBKLIPFBESYUW5W7ZTDZ6FDMG6SQ, seg_pos={0 0}, upload=56369]: Done uploading piece. err: <nil> Jul 26 22:22:11.254721792 (58h46m42.445020379s) [noise=true , address=79.127.201.215:4354 , piece_id=DQLAS4XZR645YFIQASJBE7X6HKC2XVDZV5K5US3Q76555AR3CJ5A, seg_pos={0 0}, upload=56369]: Done uploading piece. err: <nil> Jul 26 22:22:30.059804940 (58h47m1.250103522s) [noise=true , address=104.36.82.18:4393 , piece_id=FWD77PNVTHRCVR3MONVKFDMGUQWNYFMVOYTAHQINTA6MWSYVTGPQ, seg_pos={0 0}, upload=56369]: Done uploading piece. err: <nil> Jul 26 22:22:41.443136499 (58h47m12.633435085s) [noise=true , address=104.36.82.13:4193 , piece_id=DHTUE25QWQVB7KFJHWQNBXYNCGFPN5B4WLH7OTQYOMIVHLDMKCBQ, seg_pos={0 0}, upload=56369]: Done uploading piece. err: <nil> Jul 26 22:22:44.716763040 (58h47m15.907061605s) [noise=true , address=104.36.82.5:4851 , piece_id=G274D6QIAY7LUO7L63C2ZRAB4TUB3MHK5GSMVCWM7DWMWYFBVO6A, seg_pos={0 0}, upload=56369]: Done uploading piece. err: <nil> Jul 26 22:22:51.233720998 (58h47m22.424019571s) [noise=true , address=104.36.82.38:4722 , piece_id=WPL7ZECB5VUOP5XROZBAL5BBOCSDNOAWMGUAKWLX23CERGOP6ZBA, seg_pos={0 0}, upload=56369]: Done uploading piece. err: <nil> Jul 26 22:23:05.446528521 (58h47m36.636827128s) [noise=true , address=104.36.82.41:4762 , piece_id=JAKPPT76JPKAAQVETWT5Y6DKSKH6YDO6RH6BM7IK5N2GYVJALM4Q, seg_pos={0 0}, upload=56369]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:104.36.82.41:4762): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:23:05.446598972 (58h47m36.636897551s) [noise=true , address=104.36.82.12:4180 , piece_id=6X4GG65QCJKDEHONU6AIA6MPKGI6GJRNUQ22SWQIIMU7JDS362WA, seg_pos={0 0}, upload=56369]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:4180): protocol: write tcp 139.59.144.22:58762->104.36.82.12:4180: use of closed network connection; write tcp 139.59.144.22:58762->104.36.82.12:4180: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58762->104.36.82.12:4180: use of closed network connection Jul 26 22:23:05.446652963 (58h47m36.636951546s) [noise=true , address=104.36.82.39:4757 , piece_id=CPSCZA4I2LQI2MXJPJTVHFBTXKL7QRIXVTHNVL22EB3ML55ZFNWA, seg_pos={0 0}, upload=56369]: Done uploading piece. err: ecclient: upload failed (node:18M8G3TUJ9jLNDnWMpYaQBXfBUXpD7hzf4Xt6BzasYNnsfTWxS, address:104.36.82.39:4757): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:23:05.446818015 (58h47m36.637116595s) [noise=true , address=104.36.82.35:4592 , piece_id=7XRGNLGYRQ6TXZW72ROZUS2RYLVZIACFYIRQBMPCS7HEF5SNP4GA, seg_pos={0 0}, upload=56369]: Done uploading piece. err: ecclient: upload failed (node:1fB7rqiDhMNQ4j9smkctbKdnGCXXdfFPWTgTxRrwQVZ9fKEFeZ, address:104.36.82.35:4592): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:23:05.446920774 (58h47m36.637219353s) [noise=true , address=77.90.31.13:4651 , piece_id=P5VJKPWCN37UCEXBRTQZFBKHJJEGJQ7C6L3HXP7OZ4USPNIIYVHA, seg_pos={0 0}, upload=56369]: Done uploading piece. err: ecclient: upload failed (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o, address:77.90.31.13:4651): protocol: rpc: dial tcp 77.90.31.13:4651: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:05.446984862 (58h47m36.637283446s) [noise=true , address=104.36.82.34:4561 , piece_id=HIPZEOWJERVJAIGWGRP5EYFA4OFZVFQV2DEZV6M4FYRLFDA2BSQQ, seg_pos={0 0}, upload=56369]: Done uploading piece. err: ecclient: upload failed (node:12NTJdWYbLU6o4MTfyp7iPzJCmWYG288BgdGSN8JDcLyYGhCSkb, address:104.36.82.34:4561): protocol: write tcp 139.59.144.22:42898->104.36.82.34:4561: use of closed network connection; write tcp 139.59.144.22:42898->104.36.82.34:4561: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42898->104.36.82.34:4561: use of closed network connection Jul 26 22:23:05.447022362 (58h47m36.637320955s) [seg_pos={0 0}, upload=56369]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:23:05.558934620 (58h47m36.749233206s) [upload=56369]: Done uploading segments... Jul 26 22:23:05.558966378 (58h47m36.749264971s) [upload=56369]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.11503282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................XX.X..................X.X..................] duration: 2.483577028s FAIL (2025-07-26 22:23:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.483608182s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 738.025448ms FAIL (2025-07-26 22:23:03 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 738.090041ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.579855757s FAIL (2025-07-26 22:22:42 UTC) orbiter warning: context deadline exceeded Duration: 2.579879781s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.......X.] duration: 14.252460181s FAIL (2025-07-26 22:22:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.252471903s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................XX.X..................X.X.................] duration: 2.26118174s FAIL (2025-07-26 22:22:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.261219399s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.......X] duration: 1m0.001348419s FAIL (2025-07-26 22:22:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001455787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...................X.X..X..XXX....X.X.X...............X.....] duration: 2.296456364s FAIL (2025-07-26 22:22:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.296484721s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......................................X.............XX...XXX] duration: 1m0.004081582s FAIL (2025-07-26 22:22:05 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== questItem_ObjectBegin Jul 26 22:21:05.511191210 (58h45m36.701489798s) [upload=56352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:21:05.705250904 (58h45m36.895549483s) [seg_pos={0 0}, upload=56352]: Begin upload segment... Jul 26 22:21:05.705318777 (58h45m36.895617367s) [seg_pos={0 0}, upload=56352]: Done begin upload segment. Jul 26 22:21:05.705378314 (58h45m36.895676899s) [noise=true , address=79.127.205.246:4277 , piece_id=2RVTLMLPTXGRYECQU4LS2FYFZKWMRAUH7PYDSROWT7D44JVFZ7IA, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.705848027 (58h45m36.896146606s) [noise=true , address=104.36.82.36:4651 , piece_id=W6GRCDUZ2KKCKRXAN37TU2KDR4LPVDEHUCAKJJ3T2RRLHQXPAEGA, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.705986570 (58h45m36.896285152s) [noise=true , address=104.36.82.56:4230 , piece_id=HYEFYFRQ6O5PJVAL3Y46W7XSUO6YUI32LJ5MZJ3XTL6XZ4PX2FBA, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.706211354 (58h45m36.896509945s) [noise=true , address=77.90.31.12:4615 , piece_id=KKK6ANFTUOOTUP2QIUPJNKJEGFHOO7TOT4MSADMT5DE57CMDKT4A, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.706388630 (58h45m36.89668721s) [noise=true , address=104.36.82.4:4822 , piece_id=RZITTZRU7ANIA3FQZMJC7B7WPV5FIET3IDIPMROLW5B4RWVMHUUA, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.706576944 (58h45m36.896875536s) [noise=true , address=79.127.219.38:4010 , piece_id=XSIH5E2UKRJ6EN6JXC537PHYWC2ZLT2NEHSJM4TQIBRP56RJLXYQ, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.706745857 (58h45m36.897044436s) [noise=true , address=104.36.82.5:4853 , piece_id=ICPAXLOI4A4DFUN7YMGBHD2FMZBN4WNTXGRMPWZL2YGZNWKRMEEQ, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.707008075 (58h45m36.897306664s) [noise=true , address=79.127.201.215:4350 , piece_id=CCXITIEJBEJS6CQCPZLPRBYEKPDUS4NC4QNALQSPHEZY65RKGNXQ, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.707175154 (58h45m36.897473736s) [noise=true , address=104.36.82.41:4795 , piece_id=BXEPT6GWAJYVAIDKU3EV74DWAYDRNRRBDEMJY6TVXH2TOGMGQSMQ, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.707453272 (58h45m36.897751858s) [noise=true , address=79.127.219.51:4212 , piece_id=J5FLZJLHLHLPTDYGDDSUF6HZTGAO2BUMRMDJKPA4M5T22MLSA2KQ, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.707660469 (58h45m36.897959047s) [noise=true , address=64.49.13.169:4716 , piece_id=6FGO4LTO6JTCVXIBMYD5TFAJ6LL43AJO4YJ4PMERMXQ5GOTYWXGQ, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.707863880 (58h45m36.898162469s) [noise=true , address=104.36.82.39:4760 , piece_id=WYJHEJNOKSXGMBTTC2MEOQVLSK46TI3WAA5UGL626KWDZWDHZAKA, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.708010127 (58h45m36.898308774s) [noise=true , address=64.49.13.138:4483 , piece_id=7R2M4JLBG6BCVA5PXDH3LWLHUSRVPC452VKQA2RYLDOGNBM4LTLA, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.708414672 (58h45m36.898713264s) [noise=true , address=79.127.205.235:4675 , piece_id=3M3WQBTLXQSDL4TMZKGIIEBRIU55WHYMHP2HNMRAXEDFAGZS23FQ, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.708566471 (58h45m36.898865052s) [noise=true , address=64.49.13.140:4508 , piece_id=V4X6UU6FRX4ZHRTTOOPA4IOK2XIMU3IBB44NSV6PZGTD4NX53FBQ, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.708797010 (58h45m36.899095598s) [noise=true , address=104.36.82.7:4917 , piece_id=5ARMHLQD7BSKMF5DYLZP3T5NB5PAQMBSG75Y6LXCKVECX3AGUAMA, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.708963479 (58h45m36.899262059s) [noise=true , address=104.36.82.21:4498 , piece_id=EKOH5KZ3CZ3FC7F5Z3KU223F2ROIBEHXZ3JGUBIQMEKAXNUY5XFA, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:05.709348010 (58h45m36.899646592s) [noise=true , address=79.127.205.234:4603 , piece_id=HF5A6Z2FM55JG4LPC4OG7JE55F4OAXWREWP42RN2WVM63HFSN3LQ, seg_pos={0 0}, upload=56352]: Uploading piece... Jul 26 22:21:06.294093672 (58h45m37.48439226s) [upload=56352]: Next returned nil segment Jul 26 22:21:06.294130864 (58h45m37.484429458s) [upload=56352]: Waiting for error group managing segments... Jul 26 22:21:06.507326932 (58h45m37.69762551s) [noise=true , address=104.36.82.56:4230 , piece_id=HYEFYFRQ6O5PJVAL3Y46W7XSUO6YUI32LJ5MZJ3XTL6XZ4PX2FBA, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:06.891357152 (58h45m38.08165573s) [noise=true , address=64.49.13.138:4483 , piece_id=7R2M4JLBG6BCVA5PXDH3LWLHUSRVPC452VKQA2RYLDOGNBM4LTLA, seg_pos={0 0}, upload=56352]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:21:06.974769570 (58h45m38.165068136s) [noise=true , address=79.127.205.246:4277 , piece_id=2RVTLMLPTXGRYECQU4LS2FYFZKWMRAUH7PYDSROWT7D44JVFZ7IA, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:07.116072287 (58h45m38.306370867s) [noise=true , address=79.127.205.235:4675 , piece_id=3M3WQBTLXQSDL4TMZKGIIEBRIU55WHYMHP2HNMRAXEDFAGZS23FQ, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:07.150125914 (58h45m38.340424481s) [noise=true , address=79.127.219.51:4212 , piece_id=J5FLZJLHLHLPTDYGDDSUF6HZTGAO2BUMRMDJKPA4M5T22MLSA2KQ, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:07.173755116 (58h45m38.364053696s) [noise=true , address=79.127.205.234:4603 , piece_id=HF5A6Z2FM55JG4LPC4OG7JE55F4OAXWREWP42RN2WVM63HFSN3LQ, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:08.104671221 (58h45m39.294969802s) [noise=true , address=104.36.82.5:4853 , piece_id=ICPAXLOI4A4DFUN7YMGBHD2FMZBN4WNTXGRMPWZL2YGZNWKRMEEQ, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:08.669108893 (58h45m39.859407466s) [noise=true , address=64.49.13.169:4716 , piece_id=6FGO4LTO6JTCVXIBMYD5TFAJ6LL43AJO4YJ4PMERMXQ5GOTYWXGQ, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:10.750700038 (58h45m41.94099861s) [noise=true , address=79.127.219.38:4010 , piece_id=XSIH5E2UKRJ6EN6JXC537PHYWC2ZLT2NEHSJM4TQIBRP56RJLXYQ, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:11.431112398 (58h45m42.621410967s) [noise=true , address=64.49.13.140:4508 , piece_id=V4X6UU6FRX4ZHRTTOOPA4IOK2XIMU3IBB44NSV6PZGTD4NX53FBQ, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:23.838946043 (58h45m55.029244618s) [noise=true , address=104.36.82.4:4822 , piece_id=RZITTZRU7ANIA3FQZMJC7B7WPV5FIET3IDIPMROLW5B4RWVMHUUA, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:24.039121994 (58h45m55.229420563s) [noise=true , address=79.127.201.215:4350 , piece_id=CCXITIEJBEJS6CQCPZLPRBYEKPDUS4NC4QNALQSPHEZY65RKGNXQ, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:31.010067928 (58h46m2.200366512s) [noise=true , address=104.36.82.7:4917 , piece_id=5ARMHLQD7BSKMF5DYLZP3T5NB5PAQMBSG75Y6LXCKVECX3AGUAMA, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:39.565667209 (58h46m10.755965793s) [noise=true , address=77.90.31.12:4615 , piece_id=KKK6ANFTUOOTUP2QIUPJNKJEGFHOO7TOT4MSADMT5DE57CMDKT4A, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:57.460303844 (58h46m28.650602417s) [noise=true , address=104.36.82.36:4651 , piece_id=W6GRCDUZ2KKCKRXAN37TU2KDR4LPVDEHUCAKJJ3T2RRLHQXPAEGA, seg_pos={0 0}, upload=56352]: Done uploading piece. err: <nil> Jul 26 22:21:57.460340697 (58h46m28.650639277s) [seg_pos={0 0}, upload=56352]: Segment reached optimal threshold of 14 pieces. Jul 26 22:21:57.460466788 (58h46m28.650765358s) [noise=true , address=104.36.82.41:4795 , piece_id=BXEPT6GWAJYVAIDKU3EV74DWAYDRNRRBDEMJY6TVXH2TOGMGQSMQ, seg_pos={0 0}, upload=56352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ovwD5mScxxBf7RtRjAbBqRiqgAmTC8ajN3oBAXSaWrDQ1t9xk): protocol: expected piece hash; context canceled; EOF Jul 26 22:21:57.460629505 (58h46m28.650928074s) [noise=true , address=104.36.82.21:4498 , piece_id=EKOH5KZ3CZ3FC7F5Z3KU223F2ROIBEHXZ3JGUBIQMEKAXNUY5XFA, seg_pos={0 0}, upload=56352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UhkkwqK6ywvVTN6mF5kVMHwsbhUfqafUUWqTthYT5UBXvahPi): protocol: write tcp 139.59.144.22:35880->104.36.82.21:4498: use of closed network connection; write tcp 139.59.144.22:35880->104.36.82.21:4498: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35880->104.36.82.21:4498: use of closed network connection Jul 26 22:21:57.460663757 (58h46m28.650962329s) [noise=true , address=104.36.82.39:4760 , piece_id=WYJHEJNOKSXGMBTTC2MEOQVLSK46TI3WAA5UGL626KWDZWDHZAKA, seg_pos={0 0}, upload=56352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jJcppBK9zJzGedgY9KEjEA8twqtkdZcjhcnXJoN5AWi5QzR4E): protocol: write tcp 139.59.144.22:51452->104.36.82.39:4760: use of closed network connection; write tcp 139.59.144.22:51452->104.36.82.39:4760: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51452->104.36.82.39:4760: use of closed network connection Jul 26 22:21:57.460691259 (58h46m28.650989825s) [seg_pos={0 0}, upload=56352]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 26 22:21:57.460805738 (58h46m28.651104318s) [upload=56352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:21:57.460812460 (58h46m28.651111043s) [upload=56352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:21:57.605396086 (58h46m28.795694683s) [upload=56352]: Done uploading segments... Jul 26 22:21:57.605432727 (58h46m28.795731307s) [upload=56352]: Upload finished. err: <nil> Duration:* 1m0.004105759s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.468908033s FAIL (2025-07-26 22:21:53 UTC) orbiter warning: context deadline exceeded Duration: 2.468936678s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................XX.X..................X.X................] duration: 2.471757647s FAIL (2025-07-26 22:21:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.471796387s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 11.661888354s FAIL (2025-07-26 22:21:16 UTC) orbiter warning: context deadline exceeded Duration: 11.661959135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....................X.X..X..XXX....X.X.X...............X....] duration: 2.387837272s FAIL (2025-07-26 22:21:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.387863126s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.638590173s FAIL (2025-07-26 22:21:10 UTC) orbiter warning: context deadline exceeded Duration: 2.638736333s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................................X.............XX...XX] duration: 1m0.106081159s FAIL (2025-07-26 22:21:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Jul 26 22:20:05.450325093 (58h44m36.640623681s) [noise=true , address=45.91.116.5:4310 , piece_id=KM2YMJE3HGHZNALLRICJ7K6HA4AK44PTK4KOVIDNJYRCLARSNLTA, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:05.450374259 (58h44m36.640672849s) [noise=true , address=79.127.220.101:4048 , piece_id=4QOKIT3RBQ3HBTYKSDBTGI2QVD6ZZLF7XAMW7QLF5DZPCBBA2G5A, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:05.450813951 (58h44m36.641112539s) [noise=true , address=156.146.43.225:4449 , piece_id=7FFJGKUPF3EMANOSL2XVDRK2FEVY6OTNGPNBHY6NF7RZWLFVVB5A, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:05.450904395 (58h44m36.641202985s) [noise=true , address=104.36.82.19:4429 , piece_id=C5QFPJ2256DVMKWDSEMNCA3DBDATHEUFL7PMGU3QXDPF4CQSJCTQ, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:05.451268148 (58h44m36.641566738s) [noise=true , address=104.36.82.12:4159 , piece_id=2RFDXF5NSCYWLR2ENWS2AANPDON3K5U4BRPWYOSN5KYWCLST2AOQ, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:05.451399484 (58h44m36.641698076s) [noise=true , address=104.36.82.35:4585 , piece_id=XH5ZJSM7CEVRCZLZEZO2WCXI2PZDNPZGGRFATPEFY5Y3MWATVCPQ, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:05.451705717 (58h44m36.642004306s) [noise=true , address=104.36.82.41:4781 , piece_id=WALD4HRLRTL7ESGVCAZ7OPYAC6VRHLPOG77CJQQZUWDHTHEL7AFA, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:05.451881830 (58h44m36.642180413s) [noise=true , address=64.49.13.140:4508 , piece_id=SFIKBX7RSDF5RUT2AXAB3BHFA276Y5DV36OVSWTGPYICY73BAGQQ, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:05.452309890 (58h44m36.642608459s) [noise=true , address=79.127.201.215:4357 , piece_id=FEDY5RY5XXLVHPLFWV5TEDVHQCYZD2VMZKZZGBOIC2GVQHRKMEMQ, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:05.452402676 (58h44m36.642701263s) [noise=true , address=104.36.82.16:4304 , piece_id=E3O5X64SDOFGQ4XLTQNXYSVBLDFS57UYWEBSPBQBDMGSQXXTLWYQ, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:05.452861059 (58h44m36.643159645s) [noise=true , address=77.90.31.13:4644 , piece_id=HLENSRNEVKRYTXOBJ27US5EY4DCLSTLGT2VQ7PKU3FBLKFWXO32A, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:05.453295616 (58h44m36.643594209s) [noise=true , address=104.36.82.39:4754 , piece_id=UDDLE3ITUHOS5Z2A4NWCAFQNVS7CQ7MWLXIYYDN3TQ5YY5PI543A, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:05.453713372 (58h44m36.644011963s) [noise=true , address=64.49.13.138:4483 , piece_id=V2YDYICK4R2TRIHADOIAWPEMWWXPD3Z7VTLUFGZ5I45WJ55RJQKQ, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:05.454196942 (58h44m36.644495532s) [noise=true , address=104.36.82.44:4878 , piece_id=3ALW2GC7R6YP4UUEFDTOZM5B7FEWDE4KJ5FIEJTKMTKLK3D2PCKA, seg_pos={0 0}, upload=56337]: Uploading piece... Jul 26 22:20:06.030539142 (58h44m37.220837724s) [upload=56337]: Next returned nil segment Jul 26 22:20:06.030561214 (58h44m37.220859806s) [upload=56337]: Waiting for error group managing segments... Jul 26 22:20:06.168613674 (58h44m37.358912245s) [noise=true , address=77.90.31.13:4644 , piece_id=HLENSRNEVKRYTXOBJ27US5EY4DCLSTLGT2VQ7PKU3FBLKFWXO32A, seg_pos={0 0}, upload=56337]: Done uploading piece. err: <nil> Jul 26 22:20:06.199272362 (58h44m37.389570935s) [noise=true , address=77.90.31.11:4579 , piece_id=PUH5AC5ZWRSP6WUE5TE44HPIZIGP76DSJKRZWP3NAMBYRZXSV4IQ, seg_pos={0 0}, upload=56337]: Done uploading piece. err: <nil> Jul 26 22:20:06.724765450 (58h44m37.915064024s) [noise=true , address=64.49.13.138:4483 , piece_id=V2YDYICK4R2TRIHADOIAWPEMWWXPD3Z7VTLUFGZ5I45WJ55RJQKQ, seg_pos={0 0}, upload=56337]: Done uploading piece. err: <nil> Jul 26 22:20:06.740008846 (58h44m37.930307428s) [noise=true , address=79.127.205.235:4657 , piece_id=VIVCHF7G4BZF2XM5IONE2I4VZ3OBWZQJP2IAI4RLEHDH4UQHD4MA, seg_pos={0 0}, upload=56337]: Done uploading piece. err: <nil> Jul 26 22:20:06.767065869 (58h44m37.957364443s) [noise=true , address=79.127.205.245:4251 , piece_id=XFBPPS36TZ62XAXSYXSYPE5HRZMGG7O5QPM7GE3PC3OFLPHXQ3YQ, seg_pos={0 0}, upload=56337]: Done uploading piece. err: <nil> Jul 26 22:20:08.030554137 (58h44m39.220852719s) [noise=true , address=156.146.43.225:4449 , piece_id=7FFJGKUPF3EMANOSL2XVDRK2FEVY6OTNGPNBHY6NF7RZWLFVVB5A, seg_pos={0 0}, upload=56337]: Done uploading piece. err: <nil> Jul 26 22:20:08.045239814 (58h44m39.23553839s) [noise=true , address=79.127.219.48:4090 , piece_id=3ATAJVZKYCEF56LMAENXJ2JF7GOZDIAX5GLTVVEEPH3PM3VHCWVQ, seg_pos={0 0}, upload=56337]: Done uploading piece. err: <nil> Jul 26 22:20:08.806072460 (58h44m39.99637103s) [noise=true , address=64.49.13.140:4508 , piece_id=SFIKBX7RSDF5RUT2AXAB3BHFA276Y5DV36OVSWTGPYICY73BAGQQ, seg_pos={0 0}, upload=56337]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:20:09.362279153 (58h44m40.552577726s) [noise=true , address=79.127.220.101:4048 , piece_id=4QOKIT3RBQ3HBTYKSDBTGI2QVD6ZZLF7XAMW7QLF5DZPCBBA2G5A, seg_pos={0 0}, upload=56337]: Done uploading piece. err: <nil> Jul 26 22:20:10.370315522 (58h44m41.560614092s) [noise=true , address=45.91.116.5:4310 , piece_id=KM2YMJE3HGHZNALLRICJ7K6HA4AK44PTK4KOVIDNJYRCLARSNLTA, seg_pos={0 0}, upload=56337]: Done uploading piece. err: <nil> Jul 26 22:20:14.126595974 (58h44m45.316894559s) [noise=true , address=79.127.201.215:4357 , piece_id=FEDY5RY5XXLVHPLFWV5TEDVHQCYZD2VMZKZZGBOIC2GVQHRKMEMQ, seg_pos={0 0}, upload=56337]: Done uploading piece. err: <nil> Jul 26 22:21:04.984424626 (58h45m36.174723196s) [noise=true , address=104.36.82.12:4159 , piece_id=2RFDXF5NSCYWLR2ENWS2AANPDON3K5U4BRPWYOSN5KYWCLST2AOQ, seg_pos={0 0}, upload=56337]: Done uploading piece. err: ecclient: upload failed (node:1taQkf86SDCwKJaXVcfu1YcSNz8SfpAtChruvcZVFfy55qZdtX, address:104.36.82.12:4159): protocol: expected piece hash; write tcp 139.59.144.22:53804->104.36.82.12:4159: use of closed network connection; write tcp 139.59.144.22:53804->104.36.82.12:4159: use of closed network connection; EOF Jul 26 22:21:04.984483617 (58h45m36.174782185s) [noise=true , address=104.36.82.35:4585 , piece_id=XH5ZJSM7CEVRCZLZEZO2WCXI2PZDNPZGGRFATPEFY5Y3MWATVCPQ, seg_pos={0 0}, upload=56337]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:104.36.82.35:4585): protocol: write tcp 139.59.144.22:35914->104.36.82.35:4585: use of closed network connection; write tcp 139.59.144.22:35914->104.36.82.35:4585: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35914->104.36.82.35:4585: use of closed network connection Jul 26 22:21:04.984538554 (58h45m36.174837135s) [noise=true , address=104.36.82.16:4304 , piece_id=E3O5X64SDOFGQ4XLTQNXYSVBLDFS57UYWEBSPBQBDMGSQXXTLWYQ, seg_pos={0 0}, upload=56337]: Done uploading piece. err: ecclient: upload failed (node:12FNMhT2mqrLmJKfLtqnYqE7hpz4AzaCskjUp6oVg3dDgdau2Rv, address:104.36.82.16:4304): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:21:04.984575938 (58h45m36.174874508s) [noise=true , address=104.36.82.41:4781 , piece_id=WALD4HRLRTL7ESGVCAZ7OPYAC6VRHLPOG77CJQQZUWDHTHEL7AFA, seg_pos={0 0}, upload=56337]: Done uploading piece. err: ecclient: upload failed (node:1QhCkTzSogXFKLmFV4wQUcALhvm9YhCSQudLxSrvQWmepheiFi, address:104.36.82.41:4781): protocol: write tcp 139.59.144.22:37708->104.36.82.41:4781: use of closed network connection; write tcp 139.59.144.22:37708->104.36.82.41:4781: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37708->104.36.82.41:4781: use of closed network connection Jul 26 22:21:04.984618389 (58h45m36.174916959s) [noise=true , address=104.36.82.19:4429 , piece_id=C5QFPJ2256DVMKWDSEMNCA3DBDATHEUFL7PMGU3QXDPF4CQSJCTQ, seg_pos={0 0}, upload=56337]: Done uploading piece. err: ecclient: upload failed (node:1Bi1qafYFw9NukFHUJzV9M4Jd1i7YsjQMs7jXsVUn44pecVWzr, address:104.36.82.19:4429): protocol: write tcp 139.59.144.22:59646->104.36.82.19:4429: use of closed network connection; write tcp 139.59.144.22:59646->104.36.82.19:4429: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59646->104.36.82.19:4429: use of closed network connection Jul 26 22:21:04.984672728 (58h45m36.174971297s) [noise=true , address=104.36.82.39:4754 , piece_id=UDDLE3ITUHOS5Z2A4NWCAFQNVS7CQ7MWLXIYYDN3TQ5YY5PI543A, seg_pos={0 0}, upload=56337]: Done uploading piece. err: ecclient: upload failed (node:1HB1v3ZURPMSYsbHBjJs1neE7HQtLS1KmeLwbKK2sTKAZcXPhi, address:104.36.82.39:4754): protocol: write tcp 139.59.144.22:40210->104.36.82.39:4754: use of closed network connection; write tcp 139.59.144.22:40210->104.36.82.39:4754: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40210->104.36.82.39:4754: use of closed network connection Jul 26 22:21:04.984707949 (58h45m36.175006519s) [noise=true , address=104.36.82.44:4878 , piece_id=3ALW2GC7R6YP4UUEFDTOZM5B7FEWDE4KJ5FIEJTKMTKLK3D2PCKA, seg_pos={0 0}, upload=56337]: Done uploading piece. err: ecclient: upload failed (node:1wX9xRmyv3KWVyJhFmgBt9uYqDU6U768dqHagKZeKBSEnFup5y, address:104.36.82.44:4878): protocol: write tcp 139.59.144.22:50400->104.36.82.44:4878: use of closed network connection; write tcp 139.59.144.22:50400->104.36.82.44:4878: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50400->104.36.82.44:4878: use of closed network connection Jul 26 22:21:04.984757984 (58h45m36.175056563s) [seg_pos={0 0}, upload=56337]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:21:05.089009859 (58h45m36.27930843s) [upload=56337]: Done uploading segments... Jul 26 22:21:05.089037341 (58h45m36.279335908s) [upload=56337]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.106101555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....................X.X..X..XXX....X.X.X...............X...] duration: 2.358376038s FAIL (2025-07-26 22:20:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.358406061s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 4.111430943s FAIL (2025-07-26 22:20:36 UTC) orbiter warning: context deadline exceeded Duration: 4.111459877s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................XX.X..................X.X...............] duration: 2.504811005s FAIL (2025-07-26 22:20:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.504839561s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......................X.X..X..XXX....X.X.X...............X..] duration: 2.299069548s FAIL (2025-07-26 22:20:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.299099631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................X.............XX...X] duration: 1m0.108251188s FAIL (2025-07-26 22:20:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== JJ7HLMNKQDOLQZ3XPIHLSQQEVO2EMR3A, seg_pos={0 0}, upload=56325]: Uploading piece... Jul 26 22:19:05.216942002 (58h43m36.407240571s) [noise=true , address=104.36.82.52:4109 , piece_id=YQQSHOLERBLSFD7TGA2MIUEKRHTXLQBXM7VJTACVHNYTMNODT6DQ, seg_pos={0 0}, upload=56325]: Uploading piece... Jul 26 22:19:05.217227195 (58h43m36.407525778s) [noise=true , address=104.36.82.55:4220 , piece_id=HWADIERM4DZHOMA7EY3WJKAH6EL5YTU6PWJ5C7UDGFV3CQS3AQWA, seg_pos={0 0}, upload=56325]: Uploading piece... Jul 26 22:19:05.217559104 (58h43m36.407857683s) [noise=true , address=104.36.82.47:4540 , piece_id=TATWLJDWN7DGDGRSZVFS7LTQRUNBEXIG7PT5DD7TMFLJPYX2KU7Q, seg_pos={0 0}, upload=56325]: Uploading piece... Jul 26 22:19:05.217729759 (58h43m36.408028348s) [noise=true , address=104.36.82.4:4816 , piece_id=2KHURTRTIANIBOYY3OYITK6BJL2J3PQSE6BYITJYAUFSPE3IRK7A, seg_pos={0 0}, upload=56325]: Uploading piece... Jul 26 22:19:05.218192580 (58h43m36.408491168s) [noise=true , address=104.36.82.34:4554 , piece_id=BWTTWD5WMT4NIASN2WH2DEIGQYVVYXIGACW534EHNWKYK5JA5HMQ, seg_pos={0 0}, upload=56325]: Uploading piece... Jul 26 22:19:05.218581603 (58h43m36.408880192s) [noise=true , address=64.49.13.169:4716 , piece_id=RSX47W4FCVKQIK7WZDESFUPGH6KLFJVUJMWOFOHHI35P6RKKES7Q, seg_pos={0 0}, upload=56325]: Uploading piece... Jul 26 22:19:05.218838600 (58h43m36.409137186s) [noise=true , address=79.127.219.47:4069 , piece_id=M2U2A5COMSGVKKHGR2DZPKE2XQPXQCIDQ7EUVSTUANX7YQZONG4Q, seg_pos={0 0}, upload=56325]: Uploading piece... Jul 26 22:19:05.817114416 (58h43m37.007412981s) [upload=56325]: Next returned nil segment Jul 26 22:19:05.817131534 (58h43m37.007430116s) [upload=56325]: Waiting for error group managing segments... Jul 26 22:19:06.011406444 (58h43m37.201705009s) [noise=true , address=79.127.219.38:4005 , piece_id=IVEXDQDHE4LKZKHQNNPLZHHMBH7EP3BUWRKKY2QZ76G63EKDCNDA, seg_pos={0 0}, upload=56325]: Done uploading piece. err: <nil> Jul 26 22:19:06.067864673 (58h43m37.258163242s) [noise=true , address=79.127.219.47:4069 , piece_id=M2U2A5COMSGVKKHGR2DZPKE2XQPXQCIDQ7EUVSTUANX7YQZONG4Q, seg_pos={0 0}, upload=56325]: Done uploading piece. err: <nil> Jul 26 22:19:06.230147685 (58h43m37.420446264s) [noise=true , address=45.91.116.13:4175 , piece_id=5F4QOPHMMUNFLKBXHGFFBFGDTZOONY35QYSYCLJKVRTOI6RZXRUA, seg_pos={0 0}, upload=56325]: Done uploading piece. err: <nil> Jul 26 22:19:06.464878827 (58h43m37.655177397s) [noise=true , address=64.49.13.169:4716 , piece_id=RSX47W4FCVKQIK7WZDESFUPGH6KLFJVUJMWOFOHHI35P6RKKES7Q, seg_pos={0 0}, upload=56325]: Done uploading piece. err: <nil> Jul 26 22:19:06.588006885 (58h43m37.778305454s) [noise=true , address=79.127.205.234:4611 , piece_id=4WU6DHHDPTBKHIZAHSSAWECHBWWDKP2MU6PE4BYMWHHFE4OUBDEQ, seg_pos={0 0}, upload=56325]: Done uploading piece. err: <nil> Jul 26 22:19:06.814508025 (58h43m38.004806596s) [noise=true , address=79.127.205.245:4238 , piece_id=FMLUN34FAMZR7HM3W4FRE2QXPWE2JPCUULA5LW7NXNXFU52FZ4IQ, seg_pos={0 0}, upload=56325]: Done uploading piece. err: <nil> Jul 26 22:20:04.739435732 (58h44m35.929734373s) [noise=true , address=104.36.82.4:4816 , piece_id=2KHURTRTIANIBOYY3OYITK6BJL2J3PQSE6BYITJYAUFSPE3IRK7A, seg_pos={0 0}, upload=56325]: Done uploading piece. err: ecclient: upload failed (node:12mJkBoAJW49vAg52Ts4EgdmC6BoknUngs6ZqC136rGzdSdVLUy, address:104.36.82.4:4816): protocol: write tcp 139.59.144.22:55518->104.36.82.4:4816: use of closed network connection; write tcp 139.59.144.22:55518->104.36.82.4:4816: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55518->104.36.82.4:4816: use of closed network connection Jul 26 22:20:04.739555588 (58h44m35.929854169s) [noise=true , address=104.36.82.6:4902 , piece_id=ONNL2QJR34JGAMQ5R27R2ML7TW64VFD57CSLUSHTFRCBX2FVKSAA, seg_pos={0 0}, upload=56325]: Done uploading piece. err: ecclient: upload failed (node:12cXPznDWzr4KDxdxZDEoCpubdfEeMSdyyemGMhXcFTycSwatyv, address:104.36.82.6:4902): protocol: write tcp 139.59.144.22:51110->104.36.82.6:4902: use of closed network connection; write tcp 139.59.144.22:51110->104.36.82.6:4902: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51110->104.36.82.6:4902: use of closed network connection Jul 26 22:20:04.739632180 (58h44m35.929930761s) [noise=true , address=104.36.82.5:4849 , piece_id=5BUY65KN764JVFWGOJNUD6LIPM4TEG542ZIHVYIL47JVCWJB24DQ, seg_pos={0 0}, upload=56325]: Done uploading piece. err: ecclient: upload failed (node:1n2gSzVmNCEZcfVsFKibjXHs2jMr82D3SWQEykFQQ4uyhhnrj1, address:104.36.82.5:4849): protocol: write tcp 139.59.144.22:56746->104.36.82.5:4849: use of closed network connection; write tcp 139.59.144.22:56746->104.36.82.5:4849: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56746->104.36.82.5:4849: use of closed network connection Jul 26 22:20:04.739702177 (58h44m35.930000759s) [noise=true , address=104.36.82.46:4512 , piece_id=BY2EIBRQ4V4SKIX23MMKZHHVAICDPWDJANXPWLEPWONVVDDAKU7Q, seg_pos={0 0}, upload=56325]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:104.36.82.46:4512): protocol: write tcp 139.59.144.22:60808->104.36.82.46:4512: use of closed network connection; write tcp 139.59.144.22:60808->104.36.82.46:4512: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60808->104.36.82.46:4512: use of closed network connection Jul 26 22:20:04.739797146 (58h44m35.930095727s) [noise=true , address=104.36.82.55:4220 , piece_id=HWADIERM4DZHOMA7EY3WJKAH6EL5YTU6PWJ5C7UDGFV3CQS3AQWA, seg_pos={0 0}, upload=56325]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 139.59.144.22:52070->104.36.82.55:4220: use of closed network connection; write tcp 139.59.144.22:52070->104.36.82.55:4220: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52070->104.36.82.55:4220: use of closed network connection Jul 26 22:20:04.740487146 (58h44m35.930785733s) [noise=true , address=104.36.82.47:4540 , piece_id=TATWLJDWN7DGDGRSZVFS7LTQRUNBEXIG7PT5DD7TMFLJPYX2KU7Q, seg_pos={0 0}, upload=56325]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:104.36.82.47:4540): protocol: write tcp 139.59.144.22:43666->104.36.82.47:4540: use of closed network connection; write tcp 139.59.144.22:43666->104.36.82.47:4540: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43666->104.36.82.47:4540: use of closed network connection Jul 26 22:20:04.740559370 (58h44m35.93085795s) [noise=true , address=104.36.82.53:4141 , piece_id=S5AX3F5B7TRFXGKLNSCLJPQIIHENTROOZDRS7HKZFFL67HHLRDPQ, seg_pos={0 0}, upload=56325]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: write tcp 139.59.144.22:46212->104.36.82.53:4141: use of closed network connection; write tcp 139.59.144.22:46212->104.36.82.53:4141: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46212->104.36.82.53:4141: use of closed network connection Jul 26 22:20:04.740740897 (58h44m35.931039482s) [noise=true , address=104.36.82.22:4515 , piece_id=GJWW3DWD7OAGEFR6YV5JJJ7HLMNKQDOLQZ3XPIHLSQQEVO2EMR3A, seg_pos={0 0}, upload=56325]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:104.36.82.22:4515): protocol: read tcp 139.59.144.22:45256->104.36.82.22:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:04.740804356 (58h44m35.931102936s) [noise=true , address=104.36.82.19:4432 , piece_id=HDIJDZQOMBKILKT6TS35BUBRF5JMRBPRHBLGOVVMOQEGFK52WMEQ, seg_pos={0 0}, upload=56325]: Done uploading piece. err: ecclient: upload failed (node:12oh9XZaiLhaZdMyq1cx88yrrG9C9sHtH3RTA6W2Ly39t23ArSQ, address:104.36.82.19:4432): protocol: read tcp 139.59.144.22:33144->104.36.82.19:4432: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:04.740885949 (58h44m35.93118453s) [noise=true , address=104.36.82.52:4109 , piece_id=YQQSHOLERBLSFD7TGA2MIUEKRHTXLQBXM7VJTACVHNYTMNODT6DQ, seg_pos={0 0}, upload=56325]: Done uploading piece. err: ecclient: upload failed (node:1hBqAk4HrRmUKRw3RS7xUHZ4rdUMiF25jKWF7jqNrffZG8pEcj, address:104.36.82.52:4109): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:20:04.741106419 (58h44m35.931405s) [noise=true , address=104.36.82.12:4174 , piece_id=D6MGTIFIVVAGJ2ZP43WNMBGWDVBKSTDRYSK756A63LVHMOTB4RGA, seg_pos={0 0}, upload=56325]: Done uploading piece. err: ecclient: upload failed (node:12ZqPqvh4SVpDmkpVKQXZUW48CtzV9VMeMGJoDZwA7KpNfmjfow, address:104.36.82.12:4174): protocol: read tcp 139.59.144.22:36330->104.36.82.12:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:04.741332843 (58h44m35.931631425s) [noise=true , address=104.36.82.34:4554 , piece_id=BWTTWD5WMT4NIASN2WH2DEIGQYVVYXIGACW534EHNWKYK5JA5HMQ, seg_pos={0 0}, upload=56325]: Done uploading piece. err: ecclient: upload failed (node:12rSDkPXm47w73hry9W6BWwAiwNpB3cJuwedNt1JuSEEtd3Sup7, address:104.36.82.34:4554): protocol: read tcp 139.59.144.22:47790->104.36.82.34:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:04.741358051 (58h44m35.931656638s) [seg_pos={0 0}, upload=56325]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 22:20:04.845536190 (58h44m36.035834765s) [upload=56325]: Done uploading segments... Jul 26 22:20:04.845565668 (58h44m36.035864249s) [upload=56325]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.108270384s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.529355074s FAIL (2025-07-26 22:20:01 UTC) orbiter warning: context deadline exceeded Duration: 2.529401228s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 4.996977132s FAIL (2025-07-26 22:20:00 UTC) orbiter warning: context deadline exceeded Duration: 4.997003301s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......................X.X..X..XXX....X.X.X...............X.] duration: 2.348257164s FAIL (2025-07-26 22:19:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.348379666s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.642880947s FAIL (2025-07-26 22:19:42 UTC) orbiter warning: context deadline exceeded Duration: 2.642909634s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 24.212873467s FAIL (2025-07-26 22:19:33 UTC) orbiter warning: context deadline exceeded Duration: 24.212890946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................XX.X..................X.X..............] duration: 2.599041812s FAIL (2025-07-26 22:19:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.599072294s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................X.X..X..XXX....X.X.X...............X] duration: 30.110297275s FAIL (2025-07-26 22:19:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 30.110323123s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.536793543s FAIL (2025-07-26 22:19:15 UTC) orbiter warning: context deadline exceeded Duration: 2.536816262s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 49.032890083s FAIL (2025-07-26 22:19:09 UTC) orbiter warning: context deadline exceeded Duration: 49.032922621s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................XX.X..................X.X.............] duration: 2.451548018s FAIL (2025-07-26 22:19:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.451582703s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)