Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [....................X.......X..X...X.XX...X..X.XX......X.X..] duration: 27.109575428s FAIL (2025-07-26 22:49:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.109597822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.........................X.............................X..] duration: 2.247168923s FAIL (2025-07-26 22:48:45 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.247213915s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 857.042853ms FAIL (2025-07-26 22:48:38 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 857.073639ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.X.X...............X.......................................] duration: 2.26959742s FAIL (2025-07-26 22:48:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.269626789s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.X.........................X.............................X.] duration: 2.351267974s FAIL (2025-07-26 22:48:15 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.351354474s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.....................X.......X..X...X.XX...X..X.XX......X.X.] duration: 12.722939835s FAIL (2025-07-26 22:48:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.72295176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.X.X...............X......................................] duration: 2.407403094s FAIL (2025-07-26 22:48:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.407447289s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [...........X.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.] duration: 29.387866075s FAIL (2025-07-26 22:48:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [upload=56758]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:46:26.618945613 (59h10m57.809244203s) [seg_pos={0 0}, upload=56758]: Begin upload segment... Jul 26 22:46:26.619022371 (59h10m57.809320954s) [seg_pos={0 0}, upload=56758]: Done begin upload segment. Jul 26 22:46:26.619057588 (59h10m57.809356178s) [noise=true , address=79.127.205.245:4245 , piece_id=H6OP2EMMSPJLZ66RL7UYHWUCQ7LKVK4FI3TFNFFWYQJY2LFLBJAQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.619094889 (59h10m57.809393473s) [noise=true , address=79.127.205.233:4639 , piece_id=EJGFACK7644NTLT7ETQM6SKYHBP6XREJQSLRSTKC4FUZMSVUIYHA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.619532288 (59h10m57.809830868s) [noise=true , address=45.91.116.9:4267 , piece_id=3XNPDIF3UMR3UMC4MPMTFR7XROZ2EJAAPRKAFFJOGE4QJGZCGJ5Q, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.619638916 (59h10m57.809937502s) [noise=true , address=64.49.13.170:4721 , piece_id=TG5FJYDUE4UIONZRSBFQTFMEPW2FRE7JG6MIW4NIA3NFO4FL4NLQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.619854521 (59h10m57.810153101s) [noise=true , address=79.127.205.245:4251 , piece_id=UHBCGAWOXNFTSXNRJSM4HYZOSEJQ2IDR22MDRA25YWWO5ZT2JKYA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.620134441 (59h10m57.810433007s) [noise=true , address=77.90.31.17:4784 , piece_id=66NUZFPMKCMQPA4D2P4KSBMCSFME4CAY5T5I3H54VYWLSXNMY5AA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.620147506 (59h10m57.810446087s) [noise=true , address=79.127.220.101:4048 , piece_id=HLFJW4PEKUAJ5E5NRKBGUNLQVCUEWUCITPRXDCBV3DDTLX4C6OVA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.620484714 (59h10m57.810783301s) [noise=true , address=79.127.219.48:4090 , piece_id=R4XDKCDQDH4VYPQ2O2MMP5HLTHE2ZCK3ZGRE6GXJL6C7S4VNL6RA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.621009915 (59h10m57.8113085s) [noise=true , address=79.127.219.38:4006 , piece_id=W7P44CHNOA2TSSEGVXQ2TOM4IQ44IE4ZHNHMRXHRSWP43H2VXXJQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.621326960 (59h10m57.81162554s) [noise=true , address=45.91.116.14:4206 , piece_id=JGQOHIIB5SPMENXWR4OBCLBQHSHTMINVZWVRPPPFP4542I46NRQA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.621658501 (59h10m57.811957071s) [noise=true , address=79.127.219.47:4061 , piece_id=RJNPSRCY2DRJGQG5VU7HESGY4PV57E7IUP6UTH7RPV5QP6E6QJGQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.622044113 (59h10m57.812342703s) [noise=true , address=77.90.31.12:4608 , piece_id=CXOW5SY5XH276BW33NSUGODYEBP72RWVADAY3IVE4ZKSJVFYNGZQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.622077273 (59h10m57.812375839s) [noise=true , address=77.90.31.16:4743 , piece_id=GKGUV3JM35UKQ4J25QRTVMZJ4V6RO6IMATSRVD52ZSIZQ4FCHQOA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.622417025 (59h10m57.812715615s) [noise=true , address=79.127.201.215:4354 , piece_id=7QZIEUYVRPYRXPX4KTXOURQRMRLRJW3F6BVZALFNA7L2JJSVLJ2Q, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.622502009 (59h10m57.812800575s) [noise=true , address=77.90.31.21:4897 , piece_id=AXPOHOJWZG246DO4RJZEHWCDBUWEEHLL67X75BML5DDUVZJ5Y7XQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.622783011 (59h10m57.8130817s) [noise=true , address=77.90.31.13:4656 , piece_id=BELZANRV4EF44EYXJWUWLZ57BIS2YTCSJYR6EAW5VABWTHMRI22A, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.622883140 (59h10m57.813181705s) [noise=true , address=64.49.13.140:4508 , piece_id=Y4KNJ7RBZMAGAZEHOP7N4D5CKZWEZ4MZPYR5OQN7FA4YOUPFPRKQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.623141159 (59h10m57.813439799s) [noise=true , address=156.146.43.225:4448 , piece_id=DWS74C2CGOF5RKMIYSDDMWK677TKTCUL62GS7OWBXPYBXE3KMYUA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:27.021838441 (59h10m58.212137031s) [noise=true , address=79.127.219.47:4061 , piece_id=RJNPSRCY2DRJGQG5VU7HESGY4PV57E7IUP6UTH7RPV5QP6E6QJGQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:46:27.143425339 (59h10m58.333723909s) [noise=true , address=156.146.43.225:4448 , piece_id=DWS74C2CGOF5RKMIYSDDMWK677TKTCUL62GS7OWBXPYBXE3KMYUA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:46:27.202269529 (59h10m58.39256811s) [upload=56758]: Next returned nil segment Jul 26 22:46:27.202289918 (59h10m58.392588497s) [upload=56758]: Waiting for error group managing segments... Jul 26 22:46:27.340510680 (59h10m58.530809249s) [noise=true , address=77.90.31.21:4897 , piece_id=AXPOHOJWZG246DO4RJZEHWCDBUWEEHLL67X75BML5DDUVZJ5Y7XQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:27.413938451 (59h10m58.604237032s) [noise=true , address=79.127.219.38:4006 , piece_id=W7P44CHNOA2TSSEGVXQ2TOM4IQ44IE4ZHNHMRXHRSWP43H2VXXJQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:27.778216500 (59h10m58.968515074s) [noise=true , address=64.49.13.170:4721 , piece_id=TG5FJYDUE4UIONZRSBFQTFMEPW2FRE7JG6MIW4NIA3NFO4FL4NLQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:46:27.889151305 (59h10m59.079449885s) [noise=true , address=79.127.205.245:4251 , piece_id=UHBCGAWOXNFTSXNRJSM4HYZOSEJQ2IDR22MDRA25YWWO5ZT2JKYA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:27.898281756 (59h10m59.088580321s) [noise=true , address=79.127.205.245:4245 , piece_id=H6OP2EMMSPJLZ66RL7UYHWUCQ7LKVK4FI3TFNFFWYQJY2LFLBJAQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:27.927537614 (59h10m59.117836194s) [noise=true , address=79.127.205.233:4639 , piece_id=EJGFACK7644NTLT7ETQM6SKYHBP6XREJQSLRSTKC4FUZMSVUIYHA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:29.124079041 (59h11m0.31437762s) [noise=true , address=79.127.220.101:4048 , piece_id=HLFJW4PEKUAJ5E5NRKBGUNLQVCUEWUCITPRXDCBV3DDTLX4C6OVA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:30.163748332 (59h11m1.354046905s) [noise=true , address=45.91.116.9:4267 , piece_id=3XNPDIF3UMR3UMC4MPMTFR7XROZ2EJAAPRKAFFJOGE4QJGZCGJ5Q, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:30.363239260 (59h11m1.55353783s) [noise=true , address=79.127.219.48:4090 , piece_id=R4XDKCDQDH4VYPQ2O2MMP5HLTHE2ZCK3ZGRE6GXJL6C7S4VNL6RA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:35.888472980 (59h11m7.078771549s) [noise=true , address=64.49.13.140:4508 , piece_id=Y4KNJ7RBZMAGAZEHOP7N4D5CKZWEZ4MZPYR5OQN7FA4YOUPFPRKQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:37.524320217 (59h11m8.714618786s) [noise=true , address=45.91.116.14:4206 , piece_id=JGQOHIIB5SPMENXWR4OBCLBQHSHTMINVZWVRPPPFP4542I46NRQA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:47:07.044303486 (59h11m38.234602075s) [noise=true , address=79.127.201.215:4354 , piece_id=7QZIEUYVRPYRXPX4KTXOURQRMRLRJW3F6BVZALFNA7L2JJSVLJ2Q, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:47:26.154307939 (59h11m57.344606524s) [noise=true , address=77.90.31.17:4784 , piece_id=66NUZFPMKCMQPA4D2P4KSBMCSFME4CAY5T5I3H54VYWLSXNMY5AA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: write tcp 139.59.144.22:34718->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:47:26.154435583 (59h11m57.344734156s) [noise=true , address=77.90.31.12:4608 , piece_id=CXOW5SY5XH276BW33NSUGODYEBP72RWVADAY3IVE4ZKSJVFYNGZQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: write tcp 139.59.144.22:41956->77.90.31.12:4608: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:47:26.154476994 (59h11m57.344775567s) [noise=true , address=77.90.31.16:4743 , piece_id=GKGUV3JM35UKQ4J25QRTVMZJ4V6RO6IMATSRVD52ZSIZQ4FCHQOA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, address:77.90.31.16:4743): protocol: write tcp 139.59.144.22:54712->77.90.31.16:4743: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:47:26.154521568 (59h11m57.344820137s) [noise=true , address=77.90.31.13:4656 , piece_id=BELZANRV4EF44EYXJWUWLZ57BIS2YTCSJYR6EAW5VABWTHMRI22A, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: write tcp 139.59.144.22:44862->77.90.31.13:4656: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:47:26.154560669 (59h11m57.344859249s) [seg_pos={0 0}, upload=56758]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:47:26.256890457 (59h11m57.447189026s) [upload=56758]: Done uploading segments... Jul 26 22:47:26.256917517 (59h11m57.447216085s) [upload=56758]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 29.387897091s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [......................X.......X..X...X.XX...X..X.XX......X.X] duration: 1m0.001539679s FAIL (2025-07-26 22:47:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001635197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.X.........................X.............................X] duration: 1.603674491s FAIL (2025-07-26 22:47:46 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 1.603723379s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 850.094124ms FAIL (2025-07-26 22:47:41 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 850.169114ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [............X.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX] duration: 1m0.104688308s FAIL (2025-07-26 22:47:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [upload=56758]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:46:26.618945613 (59h10m57.809244203s) [seg_pos={0 0}, upload=56758]: Begin upload segment... Jul 26 22:46:26.619022371 (59h10m57.809320954s) [seg_pos={0 0}, upload=56758]: Done begin upload segment. Jul 26 22:46:26.619057588 (59h10m57.809356178s) [noise=true , address=79.127.205.245:4245 , piece_id=H6OP2EMMSPJLZ66RL7UYHWUCQ7LKVK4FI3TFNFFWYQJY2LFLBJAQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.619094889 (59h10m57.809393473s) [noise=true , address=79.127.205.233:4639 , piece_id=EJGFACK7644NTLT7ETQM6SKYHBP6XREJQSLRSTKC4FUZMSVUIYHA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.619532288 (59h10m57.809830868s) [noise=true , address=45.91.116.9:4267 , piece_id=3XNPDIF3UMR3UMC4MPMTFR7XROZ2EJAAPRKAFFJOGE4QJGZCGJ5Q, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.619638916 (59h10m57.809937502s) [noise=true , address=64.49.13.170:4721 , piece_id=TG5FJYDUE4UIONZRSBFQTFMEPW2FRE7JG6MIW4NIA3NFO4FL4NLQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.619854521 (59h10m57.810153101s) [noise=true , address=79.127.205.245:4251 , piece_id=UHBCGAWOXNFTSXNRJSM4HYZOSEJQ2IDR22MDRA25YWWO5ZT2JKYA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.620134441 (59h10m57.810433007s) [noise=true , address=77.90.31.17:4784 , piece_id=66NUZFPMKCMQPA4D2P4KSBMCSFME4CAY5T5I3H54VYWLSXNMY5AA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.620147506 (59h10m57.810446087s) [noise=true , address=79.127.220.101:4048 , piece_id=HLFJW4PEKUAJ5E5NRKBGUNLQVCUEWUCITPRXDCBV3DDTLX4C6OVA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.620484714 (59h10m57.810783301s) [noise=true , address=79.127.219.48:4090 , piece_id=R4XDKCDQDH4VYPQ2O2MMP5HLTHE2ZCK3ZGRE6GXJL6C7S4VNL6RA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.621009915 (59h10m57.8113085s) [noise=true , address=79.127.219.38:4006 , piece_id=W7P44CHNOA2TSSEGVXQ2TOM4IQ44IE4ZHNHMRXHRSWP43H2VXXJQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.621326960 (59h10m57.81162554s) [noise=true , address=45.91.116.14:4206 , piece_id=JGQOHIIB5SPMENXWR4OBCLBQHSHTMINVZWVRPPPFP4542I46NRQA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.621658501 (59h10m57.811957071s) [noise=true , address=79.127.219.47:4061 , piece_id=RJNPSRCY2DRJGQG5VU7HESGY4PV57E7IUP6UTH7RPV5QP6E6QJGQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.622044113 (59h10m57.812342703s) [noise=true , address=77.90.31.12:4608 , piece_id=CXOW5SY5XH276BW33NSUGODYEBP72RWVADAY3IVE4ZKSJVFYNGZQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.622077273 (59h10m57.812375839s) [noise=true , address=77.90.31.16:4743 , piece_id=GKGUV3JM35UKQ4J25QRTVMZJ4V6RO6IMATSRVD52ZSIZQ4FCHQOA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.622417025 (59h10m57.812715615s) [noise=true , address=79.127.201.215:4354 , piece_id=7QZIEUYVRPYRXPX4KTXOURQRMRLRJW3F6BVZALFNA7L2JJSVLJ2Q, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.622502009 (59h10m57.812800575s) [noise=true , address=77.90.31.21:4897 , piece_id=AXPOHOJWZG246DO4RJZEHWCDBUWEEHLL67X75BML5DDUVZJ5Y7XQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.622783011 (59h10m57.8130817s) [noise=true , address=77.90.31.13:4656 , piece_id=BELZANRV4EF44EYXJWUWLZ57BIS2YTCSJYR6EAW5VABWTHMRI22A, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.622883140 (59h10m57.813181705s) [noise=true , address=64.49.13.140:4508 , piece_id=Y4KNJ7RBZMAGAZEHOP7N4D5CKZWEZ4MZPYR5OQN7FA4YOUPFPRKQ, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:26.623141159 (59h10m57.813439799s) [noise=true , address=156.146.43.225:4448 , piece_id=DWS74C2CGOF5RKMIYSDDMWK677TKTCUL62GS7OWBXPYBXE3KMYUA, seg_pos={0 0}, upload=56758]: Uploading piece... Jul 26 22:46:27.021838441 (59h10m58.212137031s) [noise=true , address=79.127.219.47:4061 , piece_id=RJNPSRCY2DRJGQG5VU7HESGY4PV57E7IUP6UTH7RPV5QP6E6QJGQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:46:27.143425339 (59h10m58.333723909s) [noise=true , address=156.146.43.225:4448 , piece_id=DWS74C2CGOF5RKMIYSDDMWK677TKTCUL62GS7OWBXPYBXE3KMYUA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:46:27.202269529 (59h10m58.39256811s) [upload=56758]: Next returned nil segment Jul 26 22:46:27.202289918 (59h10m58.392588497s) [upload=56758]: Waiting for error group managing segments... Jul 26 22:46:27.340510680 (59h10m58.530809249s) [noise=true , address=77.90.31.21:4897 , piece_id=AXPOHOJWZG246DO4RJZEHWCDBUWEEHLL67X75BML5DDUVZJ5Y7XQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:27.413938451 (59h10m58.604237032s) [noise=true , address=79.127.219.38:4006 , piece_id=W7P44CHNOA2TSSEGVXQ2TOM4IQ44IE4ZHNHMRXHRSWP43H2VXXJQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:27.778216500 (59h10m58.968515074s) [noise=true , address=64.49.13.170:4721 , piece_id=TG5FJYDUE4UIONZRSBFQTFMEPW2FRE7JG6MIW4NIA3NFO4FL4NLQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:46:27.889151305 (59h10m59.079449885s) [noise=true , address=79.127.205.245:4251 , piece_id=UHBCGAWOXNFTSXNRJSM4HYZOSEJQ2IDR22MDRA25YWWO5ZT2JKYA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:27.898281756 (59h10m59.088580321s) [noise=true , address=79.127.205.245:4245 , piece_id=H6OP2EMMSPJLZ66RL7UYHWUCQ7LKVK4FI3TFNFFWYQJY2LFLBJAQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:27.927537614 (59h10m59.117836194s) [noise=true , address=79.127.205.233:4639 , piece_id=EJGFACK7644NTLT7ETQM6SKYHBP6XREJQSLRSTKC4FUZMSVUIYHA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:29.124079041 (59h11m0.31437762s) [noise=true , address=79.127.220.101:4048 , piece_id=HLFJW4PEKUAJ5E5NRKBGUNLQVCUEWUCITPRXDCBV3DDTLX4C6OVA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:30.163748332 (59h11m1.354046905s) [noise=true , address=45.91.116.9:4267 , piece_id=3XNPDIF3UMR3UMC4MPMTFR7XROZ2EJAAPRKAFFJOGE4QJGZCGJ5Q, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:30.363239260 (59h11m1.55353783s) [noise=true , address=79.127.219.48:4090 , piece_id=R4XDKCDQDH4VYPQ2O2MMP5HLTHE2ZCK3ZGRE6GXJL6C7S4VNL6RA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:35.888472980 (59h11m7.078771549s) [noise=true , address=64.49.13.140:4508 , piece_id=Y4KNJ7RBZMAGAZEHOP7N4D5CKZWEZ4MZPYR5OQN7FA4YOUPFPRKQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:46:37.524320217 (59h11m8.714618786s) [noise=true , address=45.91.116.14:4206 , piece_id=JGQOHIIB5SPMENXWR4OBCLBQHSHTMINVZWVRPPPFP4542I46NRQA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:47:07.044303486 (59h11m38.234602075s) [noise=true , address=79.127.201.215:4354 , piece_id=7QZIEUYVRPYRXPX4KTXOURQRMRLRJW3F6BVZALFNA7L2JJSVLJ2Q, seg_pos={0 0}, upload=56758]: Done uploading piece. err: <nil> Jul 26 22:47:26.154307939 (59h11m57.344606524s) [noise=true , address=77.90.31.17:4784 , piece_id=66NUZFPMKCMQPA4D2P4KSBMCSFME4CAY5T5I3H54VYWLSXNMY5AA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: write tcp 139.59.144.22:34718->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:47:26.154435583 (59h11m57.344734156s) [noise=true , address=77.90.31.12:4608 , piece_id=CXOW5SY5XH276BW33NSUGODYEBP72RWVADAY3IVE4ZKSJVFYNGZQ, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: write tcp 139.59.144.22:41956->77.90.31.12:4608: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:47:26.154476994 (59h11m57.344775567s) [noise=true , address=77.90.31.16:4743 , piece_id=GKGUV3JM35UKQ4J25QRTVMZJ4V6RO6IMATSRVD52ZSIZQ4FCHQOA, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, address:77.90.31.16:4743): protocol: write tcp 139.59.144.22:54712->77.90.31.16:4743: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:47:26.154521568 (59h11m57.344820137s) [noise=true , address=77.90.31.13:4656 , piece_id=BELZANRV4EF44EYXJWUWLZ57BIS2YTCSJYR6EAW5VABWTHMRI22A, seg_pos={0 0}, upload=56758]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: write tcp 139.59.144.22:44862->77.90.31.13:4656: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:47:26.154560669 (59h11m57.344859249s) [seg_pos={0 0}, upload=56758]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:47:26.256890457 (59h11m57.447189026s) [upload=56758]: Done uploading segments... Jul 26 22:47:26.256917517 (59h11m57.447216085s) [upload=56758]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.104712313s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 842.171323ms FAIL (2025-07-26 22:47:20 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 842.205772ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.X.X...............X.....................................] duration: 2.279502176s FAIL (2025-07-26 22:47:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.279537802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.X.........................X.............................] duration: 2.28642105s FAIL (2025-07-26 22:47:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.287086956s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 836.963175ms FAIL (2025-07-26 22:46:31 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 837.109829ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.......................X.......X..X...X.XX...X..X.XX......X.] duration: 6.795053421s FAIL (2025-07-26 22:46:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.795079598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [.............X.............XX...XXXXX.XXXX.XXXXX..XXX.....XX] duration: 1m0.109664101s FAIL (2025-07-26 22:46:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 908 (59h9m57.553582479s) [seg_pos={0 0}, upload=56741]: Begin upload segment... Jul 26 22:45:26.363345289 (59h9m57.553643858s) [seg_pos={0 0}, upload=56741]: Done begin upload segment. Jul 26 22:45:26.363384123 (59h9m57.553682733s) [noise=true , address=77.90.31.15:4717 , piece_id=SWMFKNGRFC7IWU6ZBHDOYTQIT43POASK633FSY4PMQ7M3QVYTGLQ, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.363417523 (59h9m57.553716104s) [noise=true , address=77.90.31.13:4636 , piece_id=UQDGBE2BZ64WXLN6LUH3XYNAZ73V6UA6NY4SNP3JQIA3Y7FMIPSQ, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.363482707 (59h9m57.55378128s) [noise=true , address=45.91.116.9:4288 , piece_id=7HI5UMF2H7FNDFLBRGDIOUGKMLBOTI7YTXMIXEKCGEWB6UDZCSSQ, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.363584274 (59h9m57.553882864s) [noise=true , address=45.91.116.9:4269 , piece_id=DDLYT5H3HTB53Y76J4JF5FNFEPGEQ5DON53ZPC2RQETEINJ7FXLA, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.363960642 (59h9m57.554259228s) [noise=true , address=64.49.13.140:4509 , piece_id=PDE2D6AIO4LXHHOY7YSHANOS3RUYHXBR4MEJXJVD63J3FTXABPOA, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.363986544 (59h9m57.554285129s) [noise=true , address=77.90.31.16:4743 , piece_id=SIS63M3NN4YXVFDU326DVJNZNUIY2OR6JTEVF2UNFW3QNS3PZU6Q, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.364483557 (59h9m57.554782143s) [noise=true , address=77.90.31.19:4875 , piece_id=UDBZ2LOA3YF2CAF7755T6RTO4D6WJ4IG6V2SLDGAGIB5TS2D3FWA, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.366258527 (59h9m57.556557116s) [noise=true , address=79.127.205.246:4279 , piece_id=5DEWFI7XZHCFQ6T6F2ZI7OVH7POAUKVGH246AULEAGYWTI57SKUA, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.366736804 (59h9m57.557035395s) [noise=true , address=79.127.205.234:4616 , piece_id=GYV53SMZFD3XR6S2XEOJN5ODAFJZ5G2BXVEJQMLCVXKIJPIU3BLQ, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.367217599 (59h9m57.557516187s) [noise=true , address=156.146.43.225:4434 , piece_id=CAJ5X4BUA7BJD6MMUYMF6AGM5BAP4ENJ7F3P32OS2AEB3KUXVWQQ, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.367536813 (59h9m57.557835393s) [noise=true , address=79.127.205.233:4652 , piece_id=QJPDYOKETUPC56CPEWDDCMV2MFPIOIKIHGEIJTPOROWXM5B3ZEVA, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.367899881 (59h9m57.558198466s) [noise=true , address=45.91.116.13:4175 , piece_id=4EARJ5FM7RPDQRXWL3PF2IXCYCA6YFUNOW72447HLGJAEQN4O5DA, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.368248916 (59h9m57.558547507s) [noise=true , address=45.91.116.14:4206 , piece_id=YBDWSJQ75K2SD7PZFMQISDSHH6U7T2XJMFOL34IXRR3BCQXCPXDQ, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.368652883 (59h9m57.558951687s) [noise=true , address=79.127.220.101:4048 , piece_id=LRDA6ZVOT5HLDIQLMPFBWZNFCLZVQUWOFZ7BCQEZATE64LVMSXCA, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.368970923 (59h9m57.559269509s) [noise=true , address=156.146.43.225:4449 , piece_id=ERWNF63DW42GI7Y2VMAP3V7D4AZHWOIALQ35M5EBSWQBXWLAU3LQ, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.369265462 (59h9m57.559564044s) [noise=true , address=79.127.205.235:4688 , piece_id=3ZMJF4N4UOKCI2MH2FCXOXTRLPLP24QOYXVH4ZTI2CHQFNDNWKKA, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.369590940 (59h9m57.55988955s) [noise=true , address=77.90.31.11:4579 , piece_id=KCSOKHQR66JBZTZ4W3KSWVH2HMGMJTL2FTD2AADX2CREUAIAAB5A, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.370193113 (59h9m57.560491705s) [noise=true , address=79.127.219.48:4090 , piece_id=PD54WUAKELTPKXZMP4AYLNS6JSGF24UU4NILEQLQHMBYPAHZAYQA, seg_pos={0 0}, upload=56741]: Uploading piece... Jul 26 22:45:26.879138498 (59h9m58.069437087s) [noise=true , address=156.146.43.225:4449 , piece_id=ERWNF63DW42GI7Y2VMAP3V7D4AZHWOIALQ35M5EBSWQBXWLAU3LQ, seg_pos={0 0}, upload=56741]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:26.897800352 (59h9m58.088098932s) [noise=true , address=156.146.43.225:4434 , piece_id=CAJ5X4BUA7BJD6MMUYMF6AGM5BAP4ENJ7F3P32OS2AEB3KUXVWQQ, seg_pos={0 0}, upload=56741]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:26.942147648 (59h9m58.132446219s) [upload=56741]: Next returned nil segment Jul 26 22:45:26.942178351 (59h9m58.13247693s) [upload=56741]: Waiting for error group managing segments... Jul 26 22:45:27.087020980 (59h9m58.277319551s) [noise=true , address=77.90.31.11:4579 , piece_id=KCSOKHQR66JBZTZ4W3KSWVH2HMGMJTL2FTD2AADX2CREUAIAAB5A, seg_pos={0 0}, upload=56741]: Done uploading piece. err: <nil> Jul 26 22:45:27.129018680 (59h9m58.319317258s) [noise=true , address=45.91.116.9:4269 , piece_id=DDLYT5H3HTB53Y76J4JF5FNFEPGEQ5DON53ZPC2RQETEINJ7FXLA, seg_pos={0 0}, upload=56741]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:27.166596325 (59h9m58.356894904s) [noise=true , address=45.91.116.13:4175 , piece_id=4EARJ5FM7RPDQRXWL3PF2IXCYCA6YFUNOW72447HLGJAEQN4O5DA, seg_pos={0 0}, upload=56741]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:27.623673776 (59h9m58.813972357s) [noise=true , address=79.127.205.234:4616 , piece_id=GYV53SMZFD3XR6S2XEOJN5ODAFJZ5G2BXVEJQMLCVXKIJPIU3BLQ, seg_pos={0 0}, upload=56741]: Done uploading piece. err: <nil> Jul 26 22:45:27.646190601 (59h9m58.836489181s) [noise=true , address=79.127.205.235:4688 , piece_id=3ZMJF4N4UOKCI2MH2FCXOXTRLPLP24QOYXVH4ZTI2CHQFNDNWKKA, seg_pos={0 0}, upload=56741]: Done uploading piece. err: <nil> Jul 26 22:45:27.714809273 (59h9m58.905107845s) [noise=true , address=79.127.205.233:4652 , piece_id=QJPDYOKETUPC56CPEWDDCMV2MFPIOIKIHGEIJTPOROWXM5B3ZEVA, seg_pos={0 0}, upload=56741]: Done uploading piece. err: <nil> Jul 26 22:45:27.743425000 (59h9m58.933723569s) [noise=true , address=79.127.219.48:4090 , piece_id=PD54WUAKELTPKXZMP4AYLNS6JSGF24UU4NILEQLQHMBYPAHZAYQA, seg_pos={0 0}, upload=56741]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:28.132865884 (59h9m59.323164463s) [noise=true , address=77.90.31.16:4743 , piece_id=SIS63M3NN4YXVFDU326DVJNZNUIY2OR6JTEVF2UNFW3QNS3PZU6Q, seg_pos={0 0}, upload=56741]: Done uploading piece. err: <nil> Jul 26 22:45:28.409189285 (59h9m59.599487865s) [noise=true , address=79.127.205.246:4279 , piece_id=5DEWFI7XZHCFQ6T6F2ZI7OVH7POAUKVGH246AULEAGYWTI57SKUA, seg_pos={0 0}, upload=56741]: Done uploading piece. err: <nil> Jul 26 22:45:30.985339556 (59h10m2.175638122s) [noise=true , address=77.90.31.19:4875 , piece_id=UDBZ2LOA3YF2CAF7755T6RTO4D6WJ4IG6V2SLDGAGIB5TS2D3FWA, seg_pos={0 0}, upload=56741]: Done uploading piece. err: <nil> Jul 26 22:45:31.290657864 (59h10m2.480956443s) [noise=true , address=79.127.220.101:4048 , piece_id=LRDA6ZVOT5HLDIQLMPFBWZNFCLZVQUWOFZ7BCQEZATE64LVMSXCA, seg_pos={0 0}, upload=56741]: Done uploading piece. err: <nil> Jul 26 22:45:32.979168246 (59h10m4.169466831s) [noise=true , address=45.91.116.9:4288 , piece_id=7HI5UMF2H7FNDFLBRGDIOUGKMLBOTI7YTXMIXEKCGEWB6UDZCSSQ, seg_pos={0 0}, upload=56741]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:4288): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:35.890293666 (59h10m7.080592257s) [noise=true , address=45.91.116.14:4206 , piece_id=YBDWSJQ75K2SD7PZFMQISDSHH6U7T2XJMFOL34IXRR3BCQXCPXDQ, seg_pos={0 0}, upload=56741]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:37.062899305 (59h10m8.253197877s) [noise=true , address=64.49.13.140:4509 , piece_id=PDE2D6AIO4LXHHOY7YSHANOS3RUYHXBR4MEJXJVD63J3FTXABPOA, seg_pos={0 0}, upload=56741]: Done uploading piece. err: <nil> Jul 26 22:46:25.899053169 (59h10m57.089351755s) [noise=true , address=77.90.31.13:4636 , piece_id=UQDGBE2BZ64WXLN6LUH3XYNAZ73V6UA6NY4SNP3JQIA3Y7FMIPSQ, seg_pos={0 0}, upload=56741]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: write tcp 139.59.144.22:54116->77.90.31.13:4636: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:46:25.899096733 (59h10m57.089395312s) [noise=true , address=77.90.31.15:4717 , piece_id=SWMFKNGRFC7IWU6ZBHDOYTQIT43POASK633FSY4PMQ7M3QVYTGLQ, seg_pos={0 0}, upload=56741]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 139.59.144.22:46370->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:46:25.899148306 (59h10m57.089446887s) [seg_pos={0 0}, upload=56741]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:46:26.006479912 (59h10m57.196778479s) [upload=56741]: Done uploading segments... Jul 26 22:46:26.006506622 (59h10m57.196805201s) [upload=56741]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.109766437s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [........................X.......X..X...X.XX...X..X.XX......X] duration: 1m0.000623726s FAIL (2025-07-26 22:46:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000645106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.X.........................X............................] duration: 2.378124447s FAIL (2025-07-26 22:46:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.378223599s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 824.847176ms FAIL (2025-07-26 22:46:06 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 824.903479ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.X.X...............X....................................] duration: 2.446900822s FAIL (2025-07-26 22:46:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.446937247s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.X.........................X...........................] duration: 2.24395117s FAIL (2025-07-26 22:45:42 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.243984564s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.X.X...............X...................................] duration: 2.265105666s FAIL (2025-07-26 22:45:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.265140489s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [..............X.............XX...XXXXX.XXXX.XXXXX..XXX.....X] duration: 1m0.120714713s FAIL (2025-07-26 22:45:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== upload segment... Jul 26 22:44:20.366386426 (59h8m51.556685014s) [seg_pos={0 0}, upload=56726]: Done begin upload segment. Jul 26 22:44:20.366443440 (59h8m51.556742022s) [noise=true , address=79.127.201.215:4350 , piece_id=LBSE774F3CPPFIPCEWO2JZQBOE465G6M3HKNT5FMMII565KDIANA, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.366452477 (59h8m51.556751066s) [noise=true , address=77.90.31.13:4658 , piece_id=HZGWO25EZKF6OOYAMZQRMDONXNIMAWOFVK7WYDND4P2M2JU5M2LA, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.366953783 (59h8m51.557252375s) [noise=true , address=79.127.219.48:4090 , piece_id=36ZSHYEVQGJCQTS6DIBUZYG4J7Y4M6VFLE6ZPMOY6Z2ULSKIB4UQ, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.366985003 (59h8m51.557283593s) [noise=true , address=45.91.116.8:4065 , piece_id=WITEBRUALM5OWFSQGX66YIFFPOFNE3TATBPG7RCED7ZNFNBIIPIA, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.367320989 (59h8m51.557619571s) [noise=true , address=79.127.205.245:4227 , piece_id=IH4DZLB6FOXDA4GMRGOTSLP3K26XJ7TDSG3WHJFRMX7G5YZZPRWQ, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.367390698 (59h8m51.557689285s) [noise=true , address=77.90.31.18:4815 , piece_id=6H4AAJAAVJESPEQUXBG6WX7VZDXK5TYDQMURHG7GYEGKBAQAEPFA, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.367644347 (59h8m51.557942932s) [noise=true , address=79.127.219.38:4004 , piece_id=DWUQXYATXW2YZJ2H62H6DVECLI64OOGPCYVV76A5R3GTQ7HMCUYQ, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.367795338 (59h8m51.558093928s) [noise=true , address=45.91.116.14:4222 , piece_id=NJ7YQQDR275JVW2GGRNQ5QAAJ5IZOQSIGQXDYVBD3BASJ33NSL5A, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.368089833 (59h8m51.558388417s) [noise=true , address=45.91.116.11:4124 , piece_id=G2CDMC34APBAIGO4DP4ZI3KRRCPVSQNQHLUVS2SZOUOQHNJ4DQCQ, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.368714737 (59h8m51.559013319s) [noise=true , address=77.90.31.16:4743 , piece_id=PUX2AM2KOIE3XWAD3DAAT3DLMQI577HAETMB7PPDWKN6UGRVVTUA, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.369175630 (59h8m51.55947422s) [noise=true , address=77.90.31.10:4523 , piece_id=GGOR4KQSQIZQYGMB3FP566MKZBPC522GXTWQK7EVP5S4TX2UXD6Q, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.369819321 (59h8m51.560117903s) [noise=true , address=77.90.31.15:4732 , piece_id=QSZ3JDGVX2MCS4RXDBNVMCY4PZZCXYKH3FYUZ6KO4HNZJG7OV4SA, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.370704187 (59h8m51.561002776s) [noise=true , address=77.90.31.14:4684 , piece_id=GXXFIGMQ3DXFSEKOGFWPMNK6WSDIYLCBNXPUKUJ7FBEJKDDL62MQ, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.371187235 (59h8m51.56148582s) [noise=true , address=79.127.205.235:4691 , piece_id=Q7M76WPNZWXNADQVEFLVMDUAY5O5K5U6UMJ5YFNKQRJHLGF7YWYQ, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.371498048 (59h8m51.56179663s) [noise=true , address=77.90.31.17:4792 , piece_id=CJJDIS33UKATIO3OV3YCXU63QLHU4J2X2KBS3IVLVVXAH7FI63EA, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.371839659 (59h8m51.562138248s) [noise=true , address=104.36.82.21:4503 , piece_id=C4VJUISFB3GRANJOO64PFF2RHYMZJMWHDVXVAGSYSLO3N67FD6ZQ, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.372209356 (59h8m51.562507928s) [noise=true , address=79.127.219.51:4212 , piece_id=MHPNYEAT4UZMDXUCCZG6AMRY53HJFPFQAUUMDYVAJVJJXBOIS2KQ, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.372622043 (59h8m51.562920639s) [noise=true , address=79.127.205.246:4285 , piece_id=CWEDSEX2Q4BZSDV4BPH2Z7UKT4LMBXLOLV3S3CZPA3YI2AJNKMNA, seg_pos={0 0}, upload=56726]: Uploading piece... Jul 26 22:44:20.809538143 (59h8m51.999836717s) [noise=true , address=79.127.219.51:4212 , piece_id=MHPNYEAT4UZMDXUCCZG6AMRY53HJFPFQAUUMDYVAJVJJXBOIS2KQ, seg_pos={0 0}, upload=56726]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:44:20.920267248 (59h8m52.110565831s) [noise=true , address=79.127.201.215:4350 , piece_id=LBSE774F3CPPFIPCEWO2JZQBOE465G6M3HKNT5FMMII565KDIANA, seg_pos={0 0}, upload=56726]: Done uploading piece. err: ecclient: upload failed (node:1WhZAWvHCTj5xVkUuVZtgkFB1dhFKpBPq18YMKkepFx5qxkUAr, address:79.127.201.215:4350): protocol: storage node overloaded, request limit: 2000 Jul 26 22:44:20.952107949 (59h8m52.142406535s) [upload=56726]: Next returned nil segment Jul 26 22:44:20.952146238 (59h8m52.142444826s) [upload=56726]: Waiting for error group managing segments... Jul 26 22:44:21.016231877 (59h8m52.206530476s) [noise=true , address=79.127.219.48:4090 , piece_id=36ZSHYEVQGJCQTS6DIBUZYG4J7Y4M6VFLE6ZPMOY6Z2ULSKIB4UQ, seg_pos={0 0}, upload=56726]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:44:21.137500683 (59h8m52.327799272s) [noise=true , address=79.127.219.38:4004 , piece_id=DWUQXYATXW2YZJ2H62H6DVECLI64OOGPCYVV76A5R3GTQ7HMCUYQ, seg_pos={0 0}, upload=56726]: Done uploading piece. err: <nil> Jul 26 22:44:21.638127884 (59h8m52.828426453s) [noise=true , address=79.127.205.245:4227 , piece_id=IH4DZLB6FOXDA4GMRGOTSLP3K26XJ7TDSG3WHJFRMX7G5YZZPRWQ, seg_pos={0 0}, upload=56726]: Done uploading piece. err: <nil> Jul 26 22:44:21.667809341 (59h8m52.85810792s) [noise=true , address=79.127.205.235:4691 , piece_id=Q7M76WPNZWXNADQVEFLVMDUAY5O5K5U6UMJ5YFNKQRJHLGF7YWYQ, seg_pos={0 0}, upload=56726]: Done uploading piece. err: <nil> Jul 26 22:44:21.708759772 (59h8m52.899058363s) [noise=true , address=79.127.205.246:4285 , piece_id=CWEDSEX2Q4BZSDV4BPH2Z7UKT4LMBXLOLV3S3CZPA3YI2AJNKMNA, seg_pos={0 0}, upload=56726]: Done uploading piece. err: <nil> Jul 26 22:44:22.127575339 (59h8m53.317873907s) [noise=true , address=45.91.116.8:4065 , piece_id=WITEBRUALM5OWFSQGX66YIFFPOFNE3TATBPG7RCED7ZNFNBIIPIA, seg_pos={0 0}, upload=56726]: Done uploading piece. err: ecclient: upload failed (node:17EPCQmYb816aegE9vPSoJYByMUEvhP4DUiabis2gh9QjXQC3s, address:45.91.116.8:4065): protocol: storage node overloaded, request limit: 2000 Jul 26 22:44:22.166284893 (59h8m53.356583473s) [noise=true , address=104.36.82.21:4503 , piece_id=C4VJUISFB3GRANJOO64PFF2RHYMZJMWHDVXVAGSYSLO3N67FD6ZQ, seg_pos={0 0}, upload=56726]: Done uploading piece. err: <nil> Jul 26 22:44:22.353061078 (59h8m53.543359669s) [noise=true , address=45.91.116.11:4124 , piece_id=G2CDMC34APBAIGO4DP4ZI3KRRCPVSQNQHLUVS2SZOUOQHNJ4DQCQ, seg_pos={0 0}, upload=56726]: Done uploading piece. err: ecclient: upload failed (node:1vSQgozKbq7XjSYP5dUJjkrnTkWX4EFgBkruqUmdV3ehnoPfkr, address:45.91.116.11:4124): protocol: storage node overloaded, request limit: 2000 Jul 26 22:44:27.492398647 (59h8m58.68269724s) [noise=true , address=77.90.31.14:4684 , piece_id=GXXFIGMQ3DXFSEKOGFWPMNK6WSDIYLCBNXPUKUJ7FBEJKDDL62MQ, seg_pos={0 0}, upload=56726]: Done uploading piece. err: ecclient: upload failed (node:12a7wpA6YchhigZCKC68YJzkBKwbqg1R1gBdfN8GdvrVregrK4m, address:77.90.31.14:4684): protocol: storage node overloaded, request limit: 2000 Jul 26 22:44:33.175209010 (59h9m4.365507595s) [noise=true , address=77.90.31.18:4815 , piece_id=6H4AAJAAVJESPEQUXBG6WX7VZDXK5TYDQMURHG7GYEGKBAQAEPFA, seg_pos={0 0}, upload=56726]: Done uploading piece. err: <nil> Jul 26 22:44:34.419466426 (59h9m5.609764992s) [noise=true , address=77.90.31.16:4743 , piece_id=PUX2AM2KOIE3XWAD3DAAT3DLMQI577HAETMB7PPDWKN6UGRVVTUA, seg_pos={0 0}, upload=56726]: Done uploading piece. err: <nil> Jul 26 22:44:35.435328080 (59h9m6.625626653s) [noise=true , address=45.91.116.14:4222 , piece_id=NJ7YQQDR275JVW2GGRNQ5QAAJ5IZOQSIGQXDYVBD3BASJ33NSL5A, seg_pos={0 0}, upload=56726]: Done uploading piece. err: <nil> Jul 26 22:45:19.893419695 (59h9m51.083718293s) [noise=true , address=77.90.31.10:4523 , piece_id=GGOR4KQSQIZQYGMB3FP566MKZBPC522GXTWQK7EVP5S4TX2UXD6Q, seg_pos={0 0}, upload=56726]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:45:19.893467019 (59h9m51.083765591s) [noise=true , address=77.90.31.13:4658 , piece_id=HZGWO25EZKF6OOYAMZQRMDONXNIMAWOFVK7WYDND4P2M2JU5M2LA, seg_pos={0 0}, upload=56726]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: write tcp 139.59.144.22:39374->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:45:19.893552032 (59h9m51.083850616s) [noise=true , address=77.90.31.17:4792 , piece_id=CJJDIS33UKATIO3OV3YCXU63QLHU4J2X2KBS3IVLVVXAH7FI63EA, seg_pos={0 0}, upload=56726]: Done uploading piece. err: ecclient: upload failed (node:12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, address:77.90.31.17:4792): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:45:19.893823358 (59h9m51.084121935s) [noise=true , address=77.90.31.15:4732 , piece_id=QSZ3JDGVX2MCS4RXDBNVMCY4PZZCXYKH3FYUZ6KO4HNZJG7OV4SA, seg_pos={0 0}, upload=56726]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:45:19.893875312 (59h9m51.084173894s) [seg_pos={0 0}, upload=56726]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 22:45:20.010648566 (59h9m51.200947139s) [upload=56726]: Done uploading segments... Jul 26 22:45:20.010673161 (59h9m51.20097173s) [upload=56726]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.120739231s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 856.297853ms FAIL (2025-07-26 22:44:56 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 856.33541ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X....X.X.X...............X..................................] duration: 2.341525038s FAIL (2025-07-26 22:44:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.341620968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.X.........................X..........................] duration: 3.398177761s FAIL (2025-07-26 22:44:45 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.398209942s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.........................X.......X..X...X.XX...X..X.XX......] duration: 11.044891525s FAIL (2025-07-26 22:44:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.0449053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.X.........................X.........................] duration: 2.534031732s FAIL (2025-07-26 22:44:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.534108509s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..........................X.......X..X...X.XX...X..X.XX.....] duration: 23.019748801s FAIL (2025-07-26 22:44:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.01978493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX....X.X.X...............X.................................] duration: 2.416626521s FAIL (2025-07-26 22:44:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.416661304s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 983.891692ms FAIL (2025-07-26 22:44:09 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 983.941594ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [...............X.............XX...XXXXX.XXXX.XXXXX..XXX.....] duration: 38.027002123s FAIL (2025-07-26 22:43:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6 22:38:23.917529311 (59h2m55.107827904s) [upload=56631]: Done starting upload Jul 26 22:38:23.917846807 (59h2m55.108145388s) [upload=56631]: Uploading segments... Jul 26 22:38:23.917959648 (59h2m55.108258232s) [upload=56631]: Got next segment. Inline: false Jul 26 22:38:23.917998371 (59h2m55.108296951s) [upload=56631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:38:23.918003592 (59h2m55.108302159s) [upload=56631]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:38:24.126845246 (59h2m55.317143827s) [seg_pos={0 0}, upload=56631]: Begin upload segment... Jul 26 22:38:24.126946250 (59h2m55.317244817s) [seg_pos={0 0}, upload=56631]: Done begin upload segment. Jul 26 22:38:24.126983024 (59h2m55.317281613s) [noise=true , address=79.127.205.234:4613 , piece_id=PB57BJOZXUAWC75ONCQFPIGV3F43VD5QGR5O53IUEY3SJLOYPYPQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.127171415 (59h2m55.317470008s) [noise=true , address=79.127.219.48:4090 , piece_id=TUWO3DINZR5GBD6V6UMNETD5SQORVNADN6Y7CVG7GD2VXYUPUQ5Q, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.127653251 (59h2m55.317951845s) [noise=true , address=45.91.116.10:4100 , piece_id=BB577NEP76YKWDZI3X7MZXJEVB4BGSZZLQU5AKOWVGFO5F5ZOXFQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.127862315 (59h2m55.318160904s) [noise=true , address=79.127.219.51:4210 , piece_id=WBHIJ3ETU4NQFVTVQLMCRC3KORGCP246B5TBKAVKHLA3QISKHIAA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128042586 (59h2m55.318341175s) [noise=true , address=77.90.31.21:4880 , piece_id=R2O7OT3PYZSZV3OZVXFUBOVASKM3CJTFLHMSGBYU42O4VM5IKEDQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128393003 (59h2m55.318691592s) [noise=true , address=79.127.220.101:4048 , piece_id=EAATBR22YZLUAGBQBT2TLZIMJJWKG3VQRIZZHRGCH533MDUHIK6A, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128427794 (59h2m55.318726374s) [noise=true , address=77.90.31.17:4792 , piece_id=QBKUGW647ZUJCAT3UQBS76GOYSGQFXTF2SXA3OSTRD5QMXUOAFOA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128812762 (59h2m55.319111353s) [noise=true , address=77.90.31.10:4543 , piece_id=PUEXSYUSSZ6HYO7KFSPKQTN56VD356OH65FGWDZZNJNMVYBBQ5CA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128938664 (59h2m55.319237252s) [noise=true , address=104.36.82.10:4101 , piece_id=BV3EQW57Y4G274PPJM2WS67TTIVXT2DWCOV4U6XVB2RPACQNJQUA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129160636 (59h2m55.319459225s) [noise=true , address=104.36.82.45:4925 , piece_id=OFKLTRXDEPOGW7CUKDT7U6DUCJBP7H2T3KYHQUBSD5E35IM4SGRQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129309084 (59h2m55.319607673s) [noise=true , address=77.90.31.12:4611 , piece_id=W6HPFILAQM2GUYNGFAVVVZMM3J7SPIQY4VGBFCO5U6IIYP2WLHJQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129493745 (59h2m55.319792336s) [noise=true , address=64.49.13.140:4516 , piece_id=DPN4UUJLBJ3EEQFACSZQKJCJGGSLH6WB673M4DAREN2EF3FQNWMQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129803640 (59h2m55.32010223s) [noise=true , address=64.49.13.170:4721 , piece_id=IZWGA6PE57DVSRNG6PSUMWWH5S2STZMREIQGOKCD6HI6DFZWKNXQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129817888 (59h2m55.320116478s) [noise=true , address=156.146.43.225:4448 , piece_id=XIFOWF3YDDQNP5JQS7NPYW5ARQOURRLHCEK6PU5J3AEUL43T4BYA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.130310600 (59h2m55.320609186s) [noise=true , address=45.91.116.12:4229 , piece_id=KPDDGJ5SGXZGJFWTVUE7M4MISTLEL5DIGHNPEWSMLSTUUQFEZS5Q, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.134915854 (59h2m55.325214436s) [noise=true , address=79.127.205.235:4689 , piece_id=Q7DIHD25XP42FK6CYCO3BEGD632YPLUMYPBJKEZCBHS4YZWVJVFQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.135399255 (59h2m55.325697837s) [noise=true , address=79.127.219.38:4013 , piece_id=HP5XJHQM6W5KOYDMVH2JQN7XWCRRG5ZLKGRE22HS65MFFXVNZU4Q, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.135794642 (59h2m55.32609323s) [noise=true , address=79.127.205.246:4270 , piece_id=AIM5XOGJTJ7QQ3PPDCQ25ERTFBGRJI6UXNJAG6LDOGV5CKUGJI3A, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.550256489 (59h2m55.740555068s) [noise=true , address=79.127.219.51:4210 , piece_id=WBHIJ3ETU4NQFVTVQLMCRC3KORGCP246B5TBKAVKHLA3QISKHIAA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:24.687416075 (59h2m55.877714644s) [upload=56631]: Next returned nil segment Jul 26 22:38:24.687434636 (59h2m55.877733205s) [upload=56631]: Waiting for error group managing segments... Jul 26 22:38:24.869879720 (59h2m56.060178307s) [noise=true , address=77.90.31.17:4792 , piece_id=QBKUGW647ZUJCAT3UQBS76GOYSGQFXTF2SXA3OSTRD5QMXUOAFOA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:24.870055848 (59h2m56.060354452s) [noise=true , address=77.90.31.10:4543 , piece_id=PUEXSYUSSZ6HYO7KFSPKQTN56VD356OH65FGWDZZNJNMVYBBQ5CA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:24.878082696 (59h2m56.068381283s) [noise=true , address=77.90.31.12:4611 , piece_id=W6HPFILAQM2GUYNGFAVVVZMM3J7SPIQY4VGBFCO5U6IIYP2WLHJQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:24.904762702 (59h2m56.095061281s) [noise=true , address=64.49.13.170:4721 , piece_id=IZWGA6PE57DVSRNG6PSUMWWH5S2STZMREIQGOKCD6HI6DFZWKNXQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:24.908276832 (59h2m56.098575415s) [noise=true , address=79.127.219.38:4013 , piece_id=HP5XJHQM6W5KOYDMVH2JQN7XWCRRG5ZLKGRE22HS65MFFXVNZU4Q, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:24.949670169 (59h2m56.139968738s) [noise=true , address=77.90.31.21:4880 , piece_id=R2O7OT3PYZSZV3OZVXFUBOVASKM3CJTFLHMSGBYU42O4VM5IKEDQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:25.347251577 (59h2m56.537550146s) [noise=true , address=79.127.205.234:4613 , piece_id=PB57BJOZXUAWC75ONCQFPIGV3F43VD5QGR5O53IUEY3SJLOYPYPQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:25.439562747 (59h2m56.629861316s) [noise=true , address=79.127.205.235:4689 , piece_id=Q7DIHD25XP42FK6CYCO3BEGD632YPLUMYPBJKEZCBHS4YZWVJVFQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:25.725950048 (59h2m56.916248627s) [noise=true , address=79.127.205.246:4270 , piece_id=AIM5XOGJTJ7QQ3PPDCQ25ERTFBGRJI6UXNJAG6LDOGV5CKUGJI3A, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:27.210185703 (59h2m58.400484272s) [noise=true , address=79.127.219.48:4090 , piece_id=TUWO3DINZR5GBD6V6UMNETD5SQORVNADN6Y7CVG7GD2VXYUPUQ5Q, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:27.709038123 (59h2m58.899336692s) [noise=true , address=156.146.43.225:4448 , piece_id=XIFOWF3YDDQNP5JQS7NPYW5ARQOURRLHCEK6PU5J3AEUL43T4BYA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:28.776879313 (59h2m59.967177906s) [noise=true , address=79.127.220.101:4048 , piece_id=EAATBR22YZLUAGBQBT2TLZIMJJWKG3VQRIZZHRGCH533MDUHIK6A, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:29.935859494 (59h3m1.126158078s) [noise=true , address=64.49.13.140:4516 , piece_id=DPN4UUJLBJ3EEQFACSZQKJCJGGSLH6WB673M4DAREN2EF3FQNWMQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:32.767735980 (59h3m3.95803455s) [noise=true , address=45.91.116.12:4229 , piece_id=KPDDGJ5SGXZGJFWTVUE7M4MISTLEL5DIGHNPEWSMLSTUUQFEZS5Q, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:54.325490826 (59h3m25.515789399s) [noise=true , address=104.36.82.45:4925 , piece_id=OFKLTRXDEPOGW7CUKDT7U6DUCJBP7H2T3KYHQUBSD5E35IM4SGRQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:39:23.643352570 (59h3m54.833651156s) [noise=true , address=104.36.82.10:4101 , piece_id=BV3EQW57Y4G274PPJM2WS67TTIVXT2DWCOV4U6XVB2RPACQNJQUA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 139.59.144.22:45976->104.36.82.10:4101: use of closed network connection; write tcp 139.59.144.22:45976->104.36.82.10:4101: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45976->104.36.82.10:4101: use of closed network connection Jul 26 22:39:23.643418875 (59h3m54.833717453s) [noise=true , address=45.91.116.10:4100 , piece_id=BB577NEP76YKWDZI3X7MZXJEVB4BGSZZLQU5AKOWVGFO5F5ZOXFQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: write tcp 139.59.144.22:33428->45.91.116.10:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:39:23.643454601 (59h3m54.833753171s) [seg_pos={0 0}, upload=56631]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:39:23.749096004 (59h3m54.939394577s) [upload=56631]: Done uploading segments... Jul 26 22:39:23.749123123 (59h3m54.939421691s) [upload=56631]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 38.027030639s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [...........................X.......X..X...X.XX...X..X.XX....] duration: 10.450375075s FAIL (2025-07-26 22:43:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.450392788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [............................X.......X..X...X.XX...X..X.XX...] duration: 41.023373695s FAIL (2025-07-26 22:43:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.023394885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 939.04182ms FAIL (2025-07-26 22:43:26 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 939.072674ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.X.........................X........................] duration: 2.370396643s FAIL (2025-07-26 22:43:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.370423206s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXX....X.X.X...............X................................] duration: 2.338245315s FAIL (2025-07-26 22:43:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.338301268s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 826.395508ms FAIL (2025-07-26 22:42:45 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 826.425301ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [................X.............XX...XXXXX.XXXX.XXXXX..XXX....] duration: 34.795505103s FAIL (2025-07-26 22:42:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6 22:38:23.917529311 (59h2m55.107827904s) [upload=56631]: Done starting upload Jul 26 22:38:23.917846807 (59h2m55.108145388s) [upload=56631]: Uploading segments... Jul 26 22:38:23.917959648 (59h2m55.108258232s) [upload=56631]: Got next segment. Inline: false Jul 26 22:38:23.917998371 (59h2m55.108296951s) [upload=56631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:38:23.918003592 (59h2m55.108302159s) [upload=56631]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:38:24.126845246 (59h2m55.317143827s) [seg_pos={0 0}, upload=56631]: Begin upload segment... Jul 26 22:38:24.126946250 (59h2m55.317244817s) [seg_pos={0 0}, upload=56631]: Done begin upload segment. Jul 26 22:38:24.126983024 (59h2m55.317281613s) [noise=true , address=79.127.205.234:4613 , piece_id=PB57BJOZXUAWC75ONCQFPIGV3F43VD5QGR5O53IUEY3SJLOYPYPQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.127171415 (59h2m55.317470008s) [noise=true , address=79.127.219.48:4090 , piece_id=TUWO3DINZR5GBD6V6UMNETD5SQORVNADN6Y7CVG7GD2VXYUPUQ5Q, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.127653251 (59h2m55.317951845s) [noise=true , address=45.91.116.10:4100 , piece_id=BB577NEP76YKWDZI3X7MZXJEVB4BGSZZLQU5AKOWVGFO5F5ZOXFQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.127862315 (59h2m55.318160904s) [noise=true , address=79.127.219.51:4210 , piece_id=WBHIJ3ETU4NQFVTVQLMCRC3KORGCP246B5TBKAVKHLA3QISKHIAA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128042586 (59h2m55.318341175s) [noise=true , address=77.90.31.21:4880 , piece_id=R2O7OT3PYZSZV3OZVXFUBOVASKM3CJTFLHMSGBYU42O4VM5IKEDQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128393003 (59h2m55.318691592s) [noise=true , address=79.127.220.101:4048 , piece_id=EAATBR22YZLUAGBQBT2TLZIMJJWKG3VQRIZZHRGCH533MDUHIK6A, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128427794 (59h2m55.318726374s) [noise=true , address=77.90.31.17:4792 , piece_id=QBKUGW647ZUJCAT3UQBS76GOYSGQFXTF2SXA3OSTRD5QMXUOAFOA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128812762 (59h2m55.319111353s) [noise=true , address=77.90.31.10:4543 , piece_id=PUEXSYUSSZ6HYO7KFSPKQTN56VD356OH65FGWDZZNJNMVYBBQ5CA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128938664 (59h2m55.319237252s) [noise=true , address=104.36.82.10:4101 , piece_id=BV3EQW57Y4G274PPJM2WS67TTIVXT2DWCOV4U6XVB2RPACQNJQUA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129160636 (59h2m55.319459225s) [noise=true , address=104.36.82.45:4925 , piece_id=OFKLTRXDEPOGW7CUKDT7U6DUCJBP7H2T3KYHQUBSD5E35IM4SGRQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129309084 (59h2m55.319607673s) [noise=true , address=77.90.31.12:4611 , piece_id=W6HPFILAQM2GUYNGFAVVVZMM3J7SPIQY4VGBFCO5U6IIYP2WLHJQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129493745 (59h2m55.319792336s) [noise=true , address=64.49.13.140:4516 , piece_id=DPN4UUJLBJ3EEQFACSZQKJCJGGSLH6WB673M4DAREN2EF3FQNWMQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129803640 (59h2m55.32010223s) [noise=true , address=64.49.13.170:4721 , piece_id=IZWGA6PE57DVSRNG6PSUMWWH5S2STZMREIQGOKCD6HI6DFZWKNXQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129817888 (59h2m55.320116478s) [noise=true , address=156.146.43.225:4448 , piece_id=XIFOWF3YDDQNP5JQS7NPYW5ARQOURRLHCEK6PU5J3AEUL43T4BYA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.130310600 (59h2m55.320609186s) [noise=true , address=45.91.116.12:4229 , piece_id=KPDDGJ5SGXZGJFWTVUE7M4MISTLEL5DIGHNPEWSMLSTUUQFEZS5Q, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.134915854 (59h2m55.325214436s) [noise=true , address=79.127.205.235:4689 , piece_id=Q7DIHD25XP42FK6CYCO3BEGD632YPLUMYPBJKEZCBHS4YZWVJVFQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.135399255 (59h2m55.325697837s) [noise=true , address=79.127.219.38:4013 , piece_id=HP5XJHQM6W5KOYDMVH2JQN7XWCRRG5ZLKGRE22HS65MFFXVNZU4Q, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.135794642 (59h2m55.32609323s) [noise=true , address=79.127.205.246:4270 , piece_id=AIM5XOGJTJ7QQ3PPDCQ25ERTFBGRJI6UXNJAG6LDOGV5CKUGJI3A, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.550256489 (59h2m55.740555068s) [noise=true , address=79.127.219.51:4210 , piece_id=WBHIJ3ETU4NQFVTVQLMCRC3KORGCP246B5TBKAVKHLA3QISKHIAA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:24.687416075 (59h2m55.877714644s) [upload=56631]: Next returned nil segment Jul 26 22:38:24.687434636 (59h2m55.877733205s) [upload=56631]: Waiting for error group managing segments... Jul 26 22:38:24.869879720 (59h2m56.060178307s) [noise=true , address=77.90.31.17:4792 , piece_id=QBKUGW647ZUJCAT3UQBS76GOYSGQFXTF2SXA3OSTRD5QMXUOAFOA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:24.870055848 (59h2m56.060354452s) [noise=true , address=77.90.31.10:4543 , piece_id=PUEXSYUSSZ6HYO7KFSPKQTN56VD356OH65FGWDZZNJNMVYBBQ5CA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:24.878082696 (59h2m56.068381283s) [noise=true , address=77.90.31.12:4611 , piece_id=W6HPFILAQM2GUYNGFAVVVZMM3J7SPIQY4VGBFCO5U6IIYP2WLHJQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:24.904762702 (59h2m56.095061281s) [noise=true , address=64.49.13.170:4721 , piece_id=IZWGA6PE57DVSRNG6PSUMWWH5S2STZMREIQGOKCD6HI6DFZWKNXQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:24.908276832 (59h2m56.098575415s) [noise=true , address=79.127.219.38:4013 , piece_id=HP5XJHQM6W5KOYDMVH2JQN7XWCRRG5ZLKGRE22HS65MFFXVNZU4Q, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:24.949670169 (59h2m56.139968738s) [noise=true , address=77.90.31.21:4880 , piece_id=R2O7OT3PYZSZV3OZVXFUBOVASKM3CJTFLHMSGBYU42O4VM5IKEDQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:25.347251577 (59h2m56.537550146s) [noise=true , address=79.127.205.234:4613 , piece_id=PB57BJOZXUAWC75ONCQFPIGV3F43VD5QGR5O53IUEY3SJLOYPYPQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:25.439562747 (59h2m56.629861316s) [noise=true , address=79.127.205.235:4689 , piece_id=Q7DIHD25XP42FK6CYCO3BEGD632YPLUMYPBJKEZCBHS4YZWVJVFQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:25.725950048 (59h2m56.916248627s) [noise=true , address=79.127.205.246:4270 , piece_id=AIM5XOGJTJ7QQ3PPDCQ25ERTFBGRJI6UXNJAG6LDOGV5CKUGJI3A, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:27.210185703 (59h2m58.400484272s) [noise=true , address=79.127.219.48:4090 , piece_id=TUWO3DINZR5GBD6V6UMNETD5SQORVNADN6Y7CVG7GD2VXYUPUQ5Q, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:27.709038123 (59h2m58.899336692s) [noise=true , address=156.146.43.225:4448 , piece_id=XIFOWF3YDDQNP5JQS7NPYW5ARQOURRLHCEK6PU5J3AEUL43T4BYA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:28.776879313 (59h2m59.967177906s) [noise=true , address=79.127.220.101:4048 , piece_id=EAATBR22YZLUAGBQBT2TLZIMJJWKG3VQRIZZHRGCH533MDUHIK6A, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:29.935859494 (59h3m1.126158078s) [noise=true , address=64.49.13.140:4516 , piece_id=DPN4UUJLBJ3EEQFACSZQKJCJGGSLH6WB673M4DAREN2EF3FQNWMQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:32.767735980 (59h3m3.95803455s) [noise=true , address=45.91.116.12:4229 , piece_id=KPDDGJ5SGXZGJFWTVUE7M4MISTLEL5DIGHNPEWSMLSTUUQFEZS5Q, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:54.325490826 (59h3m25.515789399s) [noise=true , address=104.36.82.45:4925 , piece_id=OFKLTRXDEPOGW7CUKDT7U6DUCJBP7H2T3KYHQUBSD5E35IM4SGRQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:39:23.643352570 (59h3m54.833651156s) [noise=true , address=104.36.82.10:4101 , piece_id=BV3EQW57Y4G274PPJM2WS67TTIVXT2DWCOV4U6XVB2RPACQNJQUA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 139.59.144.22:45976->104.36.82.10:4101: use of closed network connection; write tcp 139.59.144.22:45976->104.36.82.10:4101: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45976->104.36.82.10:4101: use of closed network connection Jul 26 22:39:23.643418875 (59h3m54.833717453s) [noise=true , address=45.91.116.10:4100 , piece_id=BB577NEP76YKWDZI3X7MZXJEVB4BGSZZLQU5AKOWVGFO5F5ZOXFQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: write tcp 139.59.144.22:33428->45.91.116.10:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:39:23.643454601 (59h3m54.833753171s) [seg_pos={0 0}, upload=56631]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:39:23.749096004 (59h3m54.939394577s) [upload=56631]: Done uploading segments... Jul 26 22:39:23.749123123 (59h3m54.939421691s) [upload=56631]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 34.795544779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 835.0644ms FAIL (2025-07-26 22:42:28 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 835.105283ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.X.........................X.......................] duration: 2.470607486s FAIL (2025-07-26 22:42:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.470636104s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.XXX....X.X.X...............X...............................] duration: 2.269291519s FAIL (2025-07-26 22:42:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.269318631s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.............................X.......X..X...X.XX...X..X.XX..] duration: 21.017601392s FAIL (2025-07-26 22:41:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.017628139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.................X.............XX...XXXXX.XXXX.XXXXX..XXX...] duration: 20.106915573s FAIL (2025-07-26 22:41:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 6 22:38:23.917529311 (59h2m55.107827904s) [upload=56631]: Done starting upload Jul 26 22:38:23.917846807 (59h2m55.108145388s) [upload=56631]: Uploading segments... Jul 26 22:38:23.917959648 (59h2m55.108258232s) [upload=56631]: Got next segment. Inline: false Jul 26 22:38:23.917998371 (59h2m55.108296951s) [upload=56631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:38:23.918003592 (59h2m55.108302159s) [upload=56631]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:38:24.126845246 (59h2m55.317143827s) [seg_pos={0 0}, upload=56631]: Begin upload segment... Jul 26 22:38:24.126946250 (59h2m55.317244817s) [seg_pos={0 0}, upload=56631]: Done begin upload segment. Jul 26 22:38:24.126983024 (59h2m55.317281613s) [noise=true , address=79.127.205.234:4613 , piece_id=PB57BJOZXUAWC75ONCQFPIGV3F43VD5QGR5O53IUEY3SJLOYPYPQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.127171415 (59h2m55.317470008s) [noise=true , address=79.127.219.48:4090 , piece_id=TUWO3DINZR5GBD6V6UMNETD5SQORVNADN6Y7CVG7GD2VXYUPUQ5Q, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.127653251 (59h2m55.317951845s) [noise=true , address=45.91.116.10:4100 , piece_id=BB577NEP76YKWDZI3X7MZXJEVB4BGSZZLQU5AKOWVGFO5F5ZOXFQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.127862315 (59h2m55.318160904s) [noise=true , address=79.127.219.51:4210 , piece_id=WBHIJ3ETU4NQFVTVQLMCRC3KORGCP246B5TBKAVKHLA3QISKHIAA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128042586 (59h2m55.318341175s) [noise=true , address=77.90.31.21:4880 , piece_id=R2O7OT3PYZSZV3OZVXFUBOVASKM3CJTFLHMSGBYU42O4VM5IKEDQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128393003 (59h2m55.318691592s) [noise=true , address=79.127.220.101:4048 , piece_id=EAATBR22YZLUAGBQBT2TLZIMJJWKG3VQRIZZHRGCH533MDUHIK6A, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128427794 (59h2m55.318726374s) [noise=true , address=77.90.31.17:4792 , piece_id=QBKUGW647ZUJCAT3UQBS76GOYSGQFXTF2SXA3OSTRD5QMXUOAFOA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128812762 (59h2m55.319111353s) [noise=true , address=77.90.31.10:4543 , piece_id=PUEXSYUSSZ6HYO7KFSPKQTN56VD356OH65FGWDZZNJNMVYBBQ5CA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.128938664 (59h2m55.319237252s) [noise=true , address=104.36.82.10:4101 , piece_id=BV3EQW57Y4G274PPJM2WS67TTIVXT2DWCOV4U6XVB2RPACQNJQUA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129160636 (59h2m55.319459225s) [noise=true , address=104.36.82.45:4925 , piece_id=OFKLTRXDEPOGW7CUKDT7U6DUCJBP7H2T3KYHQUBSD5E35IM4SGRQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129309084 (59h2m55.319607673s) [noise=true , address=77.90.31.12:4611 , piece_id=W6HPFILAQM2GUYNGFAVVVZMM3J7SPIQY4VGBFCO5U6IIYP2WLHJQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129493745 (59h2m55.319792336s) [noise=true , address=64.49.13.140:4516 , piece_id=DPN4UUJLBJ3EEQFACSZQKJCJGGSLH6WB673M4DAREN2EF3FQNWMQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129803640 (59h2m55.32010223s) [noise=true , address=64.49.13.170:4721 , piece_id=IZWGA6PE57DVSRNG6PSUMWWH5S2STZMREIQGOKCD6HI6DFZWKNXQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.129817888 (59h2m55.320116478s) [noise=true , address=156.146.43.225:4448 , piece_id=XIFOWF3YDDQNP5JQS7NPYW5ARQOURRLHCEK6PU5J3AEUL43T4BYA, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.130310600 (59h2m55.320609186s) [noise=true , address=45.91.116.12:4229 , piece_id=KPDDGJ5SGXZGJFWTVUE7M4MISTLEL5DIGHNPEWSMLSTUUQFEZS5Q, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.134915854 (59h2m55.325214436s) [noise=true , address=79.127.205.235:4689 , piece_id=Q7DIHD25XP42FK6CYCO3BEGD632YPLUMYPBJKEZCBHS4YZWVJVFQ, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.135399255 (59h2m55.325697837s) [noise=true , address=79.127.219.38:4013 , piece_id=HP5XJHQM6W5KOYDMVH2JQN7XWCRRG5ZLKGRE22HS65MFFXVNZU4Q, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.135794642 (59h2m55.32609323s) [noise=true , address=79.127.205.246:4270 , piece_id=AIM5XOGJTJ7QQ3PPDCQ25ERTFBGRJI6UXNJAG6LDOGV5CKUGJI3A, seg_pos={0 0}, upload=56631]: Uploading piece... Jul 26 22:38:24.550256489 (59h2m55.740555068s) [noise=true , address=79.127.219.51:4210 , piece_id=WBHIJ3ETU4NQFVTVQLMCRC3KORGCP246B5TBKAVKHLA3QISKHIAA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:24.687416075 (59h2m55.877714644s) [upload=56631]: Next returned nil segment Jul 26 22:38:24.687434636 (59h2m55.877733205s) [upload=56631]: Waiting for error group managing segments... Jul 26 22:38:24.869879720 (59h2m56.060178307s) [noise=true , address=77.90.31.17:4792 , piece_id=QBKUGW647ZUJCAT3UQBS76GOYSGQFXTF2SXA3OSTRD5QMXUOAFOA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:24.870055848 (59h2m56.060354452s) [noise=true , address=77.90.31.10:4543 , piece_id=PUEXSYUSSZ6HYO7KFSPKQTN56VD356OH65FGWDZZNJNMVYBBQ5CA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:24.878082696 (59h2m56.068381283s) [noise=true , address=77.90.31.12:4611 , piece_id=W6HPFILAQM2GUYNGFAVVVZMM3J7SPIQY4VGBFCO5U6IIYP2WLHJQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:24.904762702 (59h2m56.095061281s) [noise=true , address=64.49.13.170:4721 , piece_id=IZWGA6PE57DVSRNG6PSUMWWH5S2STZMREIQGOKCD6HI6DFZWKNXQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:24.908276832 (59h2m56.098575415s) [noise=true , address=79.127.219.38:4013 , piece_id=HP5XJHQM6W5KOYDMVH2JQN7XWCRRG5ZLKGRE22HS65MFFXVNZU4Q, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:24.949670169 (59h2m56.139968738s) [noise=true , address=77.90.31.21:4880 , piece_id=R2O7OT3PYZSZV3OZVXFUBOVASKM3CJTFLHMSGBYU42O4VM5IKEDQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:25.347251577 (59h2m56.537550146s) [noise=true , address=79.127.205.234:4613 , piece_id=PB57BJOZXUAWC75ONCQFPIGV3F43VD5QGR5O53IUEY3SJLOYPYPQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:25.439562747 (59h2m56.629861316s) [noise=true , address=79.127.205.235:4689 , piece_id=Q7DIHD25XP42FK6CYCO3BEGD632YPLUMYPBJKEZCBHS4YZWVJVFQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:25.725950048 (59h2m56.916248627s) [noise=true , address=79.127.205.246:4270 , piece_id=AIM5XOGJTJ7QQ3PPDCQ25ERTFBGRJI6UXNJAG6LDOGV5CKUGJI3A, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:27.210185703 (59h2m58.400484272s) [noise=true , address=79.127.219.48:4090 , piece_id=TUWO3DINZR5GBD6V6UMNETD5SQORVNADN6Y7CVG7GD2VXYUPUQ5Q, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:27.709038123 (59h2m58.899336692s) [noise=true , address=156.146.43.225:4448 , piece_id=XIFOWF3YDDQNP5JQS7NPYW5ARQOURRLHCEK6PU5J3AEUL43T4BYA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:28.776879313 (59h2m59.967177906s) [noise=true , address=79.127.220.101:4048 , piece_id=EAATBR22YZLUAGBQBT2TLZIMJJWKG3VQRIZZHRGCH533MDUHIK6A, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:29.935859494 (59h3m1.126158078s) [noise=true , address=64.49.13.140:4516 , piece_id=DPN4UUJLBJ3EEQFACSZQKJCJGGSLH6WB673M4DAREN2EF3FQNWMQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:38:32.767735980 (59h3m3.95803455s) [noise=true , address=45.91.116.12:4229 , piece_id=KPDDGJ5SGXZGJFWTVUE7M4MISTLEL5DIGHNPEWSMLSTUUQFEZS5Q, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:54.325490826 (59h3m25.515789399s) [noise=true , address=104.36.82.45:4925 , piece_id=OFKLTRXDEPOGW7CUKDT7U6DUCJBP7H2T3KYHQUBSD5E35IM4SGRQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: <nil> Jul 26 22:39:23.643352570 (59h3m54.833651156s) [noise=true , address=104.36.82.10:4101 , piece_id=BV3EQW57Y4G274PPJM2WS67TTIVXT2DWCOV4U6XVB2RPACQNJQUA, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 139.59.144.22:45976->104.36.82.10:4101: use of closed network connection; write tcp 139.59.144.22:45976->104.36.82.10:4101: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45976->104.36.82.10:4101: use of closed network connection Jul 26 22:39:23.643418875 (59h3m54.833717453s) [noise=true , address=45.91.116.10:4100 , piece_id=BB577NEP76YKWDZI3X7MZXJEVB4BGSZZLQU5AKOWVGFO5F5ZOXFQ, seg_pos={0 0}, upload=56631]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: write tcp 139.59.144.22:33428->45.91.116.10:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:39:23.643454601 (59h3m54.833753171s) [seg_pos={0 0}, upload=56631]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:39:23.749096004 (59h3m54.939394577s) [upload=56631]: Done uploading segments... Jul 26 22:39:23.749123123 (59h3m54.939421691s) [upload=56631]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.106945391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..XXX....X.X.X...............X..............................] duration: 2.304354345s FAIL (2025-07-26 22:41:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.304386444s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..............................X.......X..X...X.XX...X..X.XX.] duration: 34.680466933s FAIL (2025-07-26 22:41:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.680507063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 859.042389ms FAIL (2025-07-26 22:41:15 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 859.075556ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.X.........................X......................] duration: 2.282498534s FAIL (2025-07-26 22:41:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.282540855s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)