Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX.X..................X.X.........................X.........] duration: 2.337079421s FAIL (2025-07-26 22:33:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.337108867s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 841.242482ms FAIL (2025-07-26 22:33:12 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 841.285975ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X.X..X..XXX....X.X.X...............X...................] duration: 2.249752683s FAIL (2025-07-26 22:33:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.249785768s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...........................X.............XX...XXXXX.XXXX.XXX] duration: 1m0.108819165s FAIL (2025-07-26 22:32:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 931s) [seg_pos={0 0}, upload=56527]: Done begin upload segment. Jul 26 22:31:59.786360554 (58h56m30.976659138s) [noise=true , address=79.127.219.51:4201 , piece_id=HZQ3GK57GHZKGQUITFYBYOXDE4RV3XJASLOPVDZVDHBSOQKF44SQ, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.786398666 (58h56m30.976697252s) [noise=true , address=79.127.205.234:4607 , piece_id=F2EDADHOPL35CZHQRIQRJ6HHOHOCNKQAJGDDB5S27S6ISX634CNQ, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.786711167 (58h56m30.97700975s) [noise=true , address=64.49.13.140:4508 , piece_id=PA36F6WWP3U5UVEFBBKDQ3JL4K77DCXADV2WMTB2RYOD4ICRYJRA, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.786972530 (58h56m30.977271112s) [noise=true , address=79.127.220.101:4048 , piece_id=AGLXGMO4677BMJ424ICYZBODMSY277C4BR3RGWZD4QFBUPTQURGA, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.787000433 (58h56m30.977299021s) [noise=true , address=77.90.31.19:4874 , piece_id=MYX6BDUIUO52IGN5YJHTTPZVPLFKPRKAWU4CLT3VAQTT47SX4SVQ, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.787269082 (58h56m30.977567868s) [noise=true , address=45.91.116.14:4206 , piece_id=7FPB2KZBIM43XGK3PRH5Y7THZDEZ6Q4JY4RYTJBEQOZATP2I45GA, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.787587702 (58h56m30.977886286s) [noise=true , address=45.91.116.5:4310 , piece_id=QOMUHX7ZBGAQDQF65ADN7B5YJZJGTAGNI34XPYLUMZUWB77QIVFQ, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.787731712 (58h56m30.978030315s) [noise=true , address=79.127.219.38:4003 , piece_id=WVEZA746BQL7REVAH3DKSRGA4ZWFOH2DESYTVYHX2KII2UNO4BQQ, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.788033918 (58h56m30.978332509s) [noise=true , address=77.90.31.13:4637 , piece_id=KEMW2UCSSSSOC5AYDYMJHYNJ475EN4FUQFFOHRKG3WOU3NQA5Y3A, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.788151845 (58h56m30.978450454s) [noise=true , address=64.49.13.138:4482 , piece_id=LYOQTJBFY445OSQNJ6KPQXQCLOUXDKRAE346C5NXCCSXQLACEOGQ, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.788622532 (58h56m30.978921113s) [noise=true , address=77.90.31.21:4912 , piece_id=I4T6UOISJNHZD3SZNFFHGMGKNZM2ALT2MWMQ6CFFTZ7NHJAT6H5Q, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.788988447 (58h56m30.979287035s) [noise=true , address=79.127.201.215:4343 , piece_id=VKGV6LN5NYTKOGI7PRTIXRRQY75XGH4C3LZO6DXCK4Q3QY66XV3A, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.789305166 (58h56m30.979603755s) [noise=true , address=77.90.31.16:4748 , piece_id=MF6QE65PK7G6G2STVUV5MY2EWE4XRQHM465JDEM6GRDJIBXHKDMQ, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.789641707 (58h56m30.979940294s) [noise=true , address=45.91.116.11:4146 , piece_id=LKPHIQ7I3Y73JO3JN2CKU6I2N3Z422LCNM73CEOX2SKRATBYCKMA, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.789858010 (58h56m30.980156778s) [noise=true , address=45.91.116.13:4175 , piece_id=L7RTCGWG3B3YKE4NK7WDNN24E4CNOKQQD2FFLC7W5BIWCE6SWSEQ, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.790280146 (58h56m30.980578735s) [noise=true , address=77.90.31.18:4828 , piece_id=DGKOW3367PKIL4QOO442I35JK6LQFVTOPCCQ4UHKBN5VQKWZPCNQ, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.790641675 (58h56m30.980940264s) [noise=true , address=77.90.31.12:4622 , piece_id=SLPOOLV4NFFNHSHBLODX4ALAX7ZXLMT46FUNQNIDDMSE3KCX5QAQ, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:31:59.791005741 (58h56m30.981304328s) [noise=true , address=79.127.219.48:4075 , piece_id=AU7ENHTCTPOHM3ZAWGFAU4QZHLZJFNJDDFC2Y5YZ5TDFDOBY3KTQ, seg_pos={0 0}, upload=56527]: Uploading piece... Jul 26 22:32:00.201650893 (58h56m31.391949472s) [noise=true , address=79.127.219.48:4075 , piece_id=AU7ENHTCTPOHM3ZAWGFAU4QZHLZJFNJDDFC2Y5YZ5TDFDOBY3KTQ, seg_pos={0 0}, upload=56527]: Done uploading piece. err: ecclient: upload failed (node:1E81UgrSUNzM4VMNvunqWq5dzG3WbCEU4Ke1Kf2mh1FQiK8PQR, address:79.127.219.48:4075): protocol: storage node overloaded, request limit: 2000 Jul 26 22:32:00.319917531 (58h56m31.510216123s) [noise=true , address=79.127.220.101:4048 , piece_id=AGLXGMO4677BMJ424ICYZBODMSY277C4BR3RGWZD4QFBUPTQURGA, seg_pos={0 0}, upload=56527]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:32:00.351441482 (58h56m31.541740071s) [upload=56527]: Next returned nil segment Jul 26 22:32:00.351484848 (58h56m31.541783443s) [upload=56527]: Waiting for error group managing segments... Jul 26 22:32:00.492708861 (58h56m31.683007434s) [noise=true , address=77.90.31.13:4637 , piece_id=KEMW2UCSSSSOC5AYDYMJHYNJ475EN4FUQFFOHRKG3WOU3NQA5Y3A, seg_pos={0 0}, upload=56527]: Done uploading piece. err: <nil> Jul 26 22:32:00.502716917 (58h56m31.693015498s) [noise=true , address=79.127.219.38:4003 , piece_id=WVEZA746BQL7REVAH3DKSRGA4ZWFOH2DESYTVYHX2KII2UNO4BQQ, seg_pos={0 0}, upload=56527]: Done uploading piece. err: <nil> Jul 26 22:32:00.524028070 (58h56m31.714326642s) [noise=true , address=77.90.31.21:4912 , piece_id=I4T6UOISJNHZD3SZNFFHGMGKNZM2ALT2MWMQ6CFFTZ7NHJAT6H5Q, seg_pos={0 0}, upload=56527]: Done uploading piece. err: <nil> Jul 26 22:32:00.544837252 (58h56m31.735135827s) [noise=true , address=64.49.13.140:4508 , piece_id=PA36F6WWP3U5UVEFBBKDQ3JL4K77DCXADV2WMTB2RYOD4ICRYJRA, seg_pos={0 0}, upload=56527]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:32:00.552217799 (58h56m31.742516385s) [noise=true , address=77.90.31.12:4622 , piece_id=SLPOOLV4NFFNHSHBLODX4ALAX7ZXLMT46FUNQNIDDMSE3KCX5QAQ, seg_pos={0 0}, upload=56527]: Done uploading piece. err: <nil> Jul 26 22:32:01.108502381 (58h56m32.298800967s) [noise=true , address=79.127.205.234:4607 , piece_id=F2EDADHOPL35CZHQRIQRJ6HHOHOCNKQAJGDDB5S27S6ISX634CNQ, seg_pos={0 0}, upload=56527]: Done uploading piece. err: <nil> Jul 26 22:32:01.723060462 (58h56m32.913359043s) [noise=true , address=64.49.13.138:4482 , piece_id=LYOQTJBFY445OSQNJ6KPQXQCLOUXDKRAE346C5NXCCSXQLACEOGQ, seg_pos={0 0}, upload=56527]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:32:02.216831586 (58h56m33.407130156s) [noise=true , address=79.127.219.51:4201 , piece_id=HZQ3GK57GHZKGQUITFYBYOXDE4RV3XJASLOPVDZVDHBSOQKF44SQ, seg_pos={0 0}, upload=56527]: Done uploading piece. err: <nil> Jul 26 22:32:03.646116181 (58h56m34.836414751s) [noise=true , address=45.91.116.5:4310 , piece_id=QOMUHX7ZBGAQDQF65ADN7B5YJZJGTAGNI34XPYLUMZUWB77QIVFQ, seg_pos={0 0}, upload=56527]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:32:04.429546321 (58h56m35.619844907s) [noise=true , address=45.91.116.11:4146 , piece_id=LKPHIQ7I3Y73JO3JN2CKU6I2N3Z422LCNM73CEOX2SKRATBYCKMA, seg_pos={0 0}, upload=56527]: Done uploading piece. err: <nil> Jul 26 22:32:10.471113530 (58h56m41.661412114s) [noise=true , address=79.127.201.215:4343 , piece_id=VKGV6LN5NYTKOGI7PRTIXRRQY75XGH4C3LZO6DXCK4Q3QY66XV3A, seg_pos={0 0}, upload=56527]: Done uploading piece. err: <nil> Jul 26 22:32:12.253830760 (58h56m43.44412934s) [noise=true , address=45.91.116.13:4175 , piece_id=L7RTCGWG3B3YKE4NK7WDNN24E4CNOKQQD2FFLC7W5BIWCE6SWSEQ, seg_pos={0 0}, upload=56527]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: storage node overloaded, request limit: 2000 Jul 26 22:32:32.598212343 (58h57m3.788510929s) [noise=true , address=45.91.116.14:4206 , piece_id=7FPB2KZBIM43XGK3PRH5Y7THZDEZ6Q4JY4RYTJBEQOZATP2I45GA, seg_pos={0 0}, upload=56527]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:32:59.316132370 (58h57m30.506430949s) [noise=true , address=77.90.31.19:4874 , piece_id=MYX6BDUIUO52IGN5YJHTTPZVPLFKPRKAWU4CLT3VAQTT47SX4SVQ, seg_pos={0 0}, upload=56527]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:32:59.316323225 (58h57m30.506621809s) [noise=true , address=77.90.31.18:4828 , piece_id=DGKOW3367PKIL4QOO442I35JK6LQFVTOPCCQ4UHKBN5VQKWZPCNQ, seg_pos={0 0}, upload=56527]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: rpc: dial tcp 77.90.31.18:4828: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:32:59.316549251 (58h57m30.506847822s) [noise=true , address=77.90.31.16:4748 , piece_id=MF6QE65PK7G6G2STVUV5MY2EWE4XRQHM465JDEM6GRDJIBXHKDMQ, seg_pos={0 0}, upload=56527]: Done uploading piece. err: ecclient: upload failed (node:12k122BWfbKgBxSs8MCfNFkZYh2WfZDDHLktLQRiifkPr95G7SY, address:77.90.31.16:4748): protocol: rpc: dial tcp 77.90.31.16:4748: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:32:59.316589559 (58h57m30.50688814s) [seg_pos={0 0}, upload=56527]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 22:32:59.422972405 (58h57m30.613270974s) [upload=56527]: Done uploading segments... Jul 26 22:32:59.423013809 (58h57m30.613312375s) [upload=56527]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.108839142s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX.X..................X.X.........................X........] duration: 2.296621147s FAIL (2025-07-26 22:32:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.296658586s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 826.170831ms FAIL (2025-07-26 22:32:36 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 826.287648ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X.X..X..XXX....X.X.X...............X..................] duration: 2.356676676s FAIL (2025-07-26 22:32:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.356709304s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........................................X.......X..X...X.XX.] duration: 7.467869874s FAIL (2025-07-26 22:32:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.467887939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX.X..................X.X.........................X.......] duration: 2.292809563s FAIL (2025-07-26 22:32:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.29284133s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........................................X.......X..X...X.XX] duration: 1m0.001359052s FAIL (2025-07-26 22:32:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001511541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 904.146723ms FAIL (2025-07-26 22:32:04 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 904.172671ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [............................X.............XX...XXXXX.XXXX.XX] duration: 1m0.11068316s FAIL (2025-07-26 22:31:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 067614s) [upload=56512]: Done starting upload Jul 26 22:30:59.329848376 (58h55m30.520146947s) [upload=56512]: Uploading segments... Jul 26 22:30:59.330171236 (58h55m30.520469826s) [upload=56512]: Got next segment. Inline: false Jul 26 22:30:59.330228249 (58h55m30.520526839s) [upload=56512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:30:59.330236812 (58h55m30.520535401s) [upload=56512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:30:59.522810027 (58h55m30.713108622s) [seg_pos={0 0}, upload=56512]: Begin upload segment... Jul 26 22:30:59.522895784 (58h55m30.713194373s) [seg_pos={0 0}, upload=56512]: Done begin upload segment. Jul 26 22:30:59.522928001 (58h55m30.71322659s) [noise=true , address=77.90.31.16:4765 , piece_id=OU36US5WMWIWULNGPTNGDXOQSRW7N7CLXCCHJ2YVISL2LHFYNA3Q, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.523477486 (58h55m30.713776079s) [noise=true , address=77.90.31.15:4715 , piece_id=FQIE3NHW2XY4GRTQPGJZOWJ6ZXNZ3WZKBL72252X3GVEO7QY2ZVQ, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.524037065 (58h55m30.714335656s) [noise=true , address=45.91.116.5:4310 , piece_id=TFJUCA73LCRY2H3IVQIWHHRRLOQSQWHKWWDQVIUHJWE3J7DHX52Q, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.524922618 (58h55m30.715221208s) [noise=true , address=45.91.116.14:4222 , piece_id=5BLQDHWUX5E33XHSBLI2VIG6UZUCHCEMPRJIXVY3RREM45UBPSUQ, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.525415013 (58h55m30.715713603s) [noise=true , address=45.91.116.11:4121 , piece_id=UOTAP5AUP2WJDNO77DUJMO7T37EHSS2TPMVZ6RCUJRJRBS6NLDOQ, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.525792211 (58h55m30.716090803s) [noise=true , address=79.127.205.235:4686 , piece_id=ZESJSNDJZ6J7GZXRMU4DHRBPPEPGJU37CSB7GX2WOU7UOXQRYIFA, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.526332402 (58h55m30.716630985s) [noise=true , address=79.127.220.101:4048 , piece_id=CFCTQHGC3TE6Y42IJ44Y33AT2AL6X54USDSK5XPYSKUCDMVKBIJQ, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.526776504 (58h55m30.717075086s) [noise=true , address=79.127.201.215:4351 , piece_id=EYZW7N5D7ASEIQ7HN7FPVMAEX272IAWYQ4JZFKZ4VQLA3CC4PJOA, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.527140798 (58h55m30.717439381s) [noise=true , address=79.127.205.246:4258 , piece_id=BUTWGVPOZB6XTQBZGZEM7NFA4HPI3DL5UOYQON4GLVBYJ5H66NLQ, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.527477508 (58h55m30.717776088s) [noise=true , address=156.146.43.225:4449 , piece_id=GOOZ7R2SGTLNEIRPEW35MWPIW4CQHHBOP2QH3EZ6IRT6S2NZJWAA, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.527804713 (58h55m30.718103292s) [noise=true , address=64.49.13.140:4516 , piece_id=OYXIYTR2ENC4I2TKYVH3YCU25BA244BSFZZICGUPMHCLWIZNGBBQ, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.528121010 (58h55m30.71841959s) [noise=true , address=64.49.13.138:4482 , piece_id=HOWYBGELHLSLKPW35Z4R2FUH7C6TU5YNUVCDKQ4HS5EA7O2X32QA, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.528615925 (58h55m30.718914508s) [noise=true , address=77.90.31.13:4649 , piece_id=SZUS6Y3OGTYC6JT3LYBUK4G4IMKOC5NM6SQF3ELDMNIYU3HZE3SA, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.529028085 (58h55m30.719326667s) [noise=true , address=45.91.116.12:4229 , piece_id=N6DGMU6BBYCR34F4XEDOKEYOEZRKZOLXDEPBABPGU5DA4RT2NXAQ, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.529438483 (58h55m30.719737065s) [noise=true , address=79.127.205.234:4620 , piece_id=XITKXUXPPFWKHMXBAM6DMJL6HG3AWAZ26ZGI2NL7LF5QAHH3ALQQ, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.529806968 (58h55m30.720105555s) [noise=true , address=79.127.219.38:4005 , piece_id=OZ4UDCWEEYODSLX6VDXZEV34IWLJINICIK2J5LT7G7X77IJVD7UA, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.530100571 (58h55m30.720399153s) [noise=true , address=45.91.116.8:4065 , piece_id=6DZYXGQR5ZK3QWORNC7OQDVDDAADG7WC4CH7GB4CTN2NCR54IBKQ, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:30:59.530364941 (58h55m30.720663528s) [noise=true , address=79.127.205.245:4242 , piece_id=EXQX2ZBEED2XDCLVKWFQDVBN2BPP4U5ETACPBHEREEXURKRVWAIA, seg_pos={0 0}, upload=56512]: Uploading piece... Jul 26 22:31:00.031305835 (58h55m31.221604428s) [noise=true , address=79.127.220.101:4048 , piece_id=CFCTQHGC3TE6Y42IJ44Y33AT2AL6X54USDSK5XPYSKUCDMVKBIJQ, seg_pos={0 0}, upload=56512]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:00.116105040 (58h55m31.306403633s) [upload=56512]: Next returned nil segment Jul 26 22:31:00.116131203 (58h55m31.306429796s) [upload=56512]: Waiting for error group managing segments... Jul 26 22:31:00.153264672 (58h55m31.343563243s) [noise=true , address=156.146.43.225:4449 , piece_id=GOOZ7R2SGTLNEIRPEW35MWPIW4CQHHBOP2QH3EZ6IRT6S2NZJWAA, seg_pos={0 0}, upload=56512]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:00.264997808 (58h55m31.455296378s) [noise=true , address=77.90.31.13:4649 , piece_id=SZUS6Y3OGTYC6JT3LYBUK4G4IMKOC5NM6SQF3ELDMNIYU3HZE3SA, seg_pos={0 0}, upload=56512]: Done uploading piece. err: <nil> Jul 26 22:31:00.291153300 (58h55m31.481451903s) [noise=true , address=79.127.219.38:4005 , piece_id=OZ4UDCWEEYODSLX6VDXZEV34IWLJINICIK2J5LT7G7X77IJVD7UA, seg_pos={0 0}, upload=56512]: Done uploading piece. err: <nil> Jul 26 22:31:00.538893758 (58h55m31.729192329s) [noise=true , address=45.91.116.11:4121 , piece_id=UOTAP5AUP2WJDNO77DUJMO7T37EHSS2TPMVZ6RCUJRJRBS6NLDOQ, seg_pos={0 0}, upload=56512]: Done uploading piece. err: <nil> Jul 26 22:31:00.802575858 (58h55m31.992874439s) [noise=true , address=79.127.205.235:4686 , piece_id=ZESJSNDJZ6J7GZXRMU4DHRBPPEPGJU37CSB7GX2WOU7UOXQRYIFA, seg_pos={0 0}, upload=56512]: Done uploading piece. err: <nil> Jul 26 22:31:00.831705453 (58h55m32.022004046s) [noise=true , address=79.127.205.245:4242 , piece_id=EXQX2ZBEED2XDCLVKWFQDVBN2BPP4U5ETACPBHEREEXURKRVWAIA, seg_pos={0 0}, upload=56512]: Done uploading piece. err: <nil> Jul 26 22:31:00.836161869 (58h55m32.02646085s) [noise=true , address=79.127.205.234:4620 , piece_id=XITKXUXPPFWKHMXBAM6DMJL6HG3AWAZ26ZGI2NL7LF5QAHH3ALQQ, seg_pos={0 0}, upload=56512]: Done uploading piece. err: <nil> Jul 26 22:31:00.850509548 (58h55m32.040808129s) [noise=true , address=79.127.205.246:4258 , piece_id=BUTWGVPOZB6XTQBZGZEM7NFA4HPI3DL5UOYQON4GLVBYJ5H66NLQ, seg_pos={0 0}, upload=56512]: Done uploading piece. err: <nil> Jul 26 22:31:01.058854136 (58h55m32.249152718s) [noise=true , address=64.49.13.140:4516 , piece_id=OYXIYTR2ENC4I2TKYVH3YCU25BA244BSFZZICGUPMHCLWIZNGBBQ, seg_pos={0 0}, upload=56512]: Done uploading piece. err: <nil> Jul 26 22:31:03.396665438 (58h55m34.586964018s) [noise=true , address=64.49.13.138:4482 , piece_id=HOWYBGELHLSLKPW35Z4R2FUH7C6TU5YNUVCDKQ4HS5EA7O2X32QA, seg_pos={0 0}, upload=56512]: Done uploading piece. err: <nil> Jul 26 22:31:04.481234902 (58h55m35.671533468s) [noise=true , address=45.91.116.8:4065 , piece_id=6DZYXGQR5ZK3QWORNC7OQDVDDAADG7WC4CH7GB4CTN2NCR54IBKQ, seg_pos={0 0}, upload=56512]: Done uploading piece. err: <nil> Jul 26 22:31:08.351896441 (58h55m39.542195019s) [noise=true , address=45.91.116.12:4229 , piece_id=N6DGMU6BBYCR34F4XEDOKEYOEZRKZOLXDEPBABPGU5DA4RT2NXAQ, seg_pos={0 0}, upload=56512]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:08.761533424 (58h55m39.951831993s) [noise=true , address=45.91.116.5:4310 , piece_id=TFJUCA73LCRY2H3IVQIWHHRRLOQSQWHKWWDQVIUHJWE3J7DHX52Q, seg_pos={0 0}, upload=56512]: Done uploading piece. err: <nil> Jul 26 22:31:19.683639221 (58h55m50.87393779s) [noise=true , address=45.91.116.14:4222 , piece_id=5BLQDHWUX5E33XHSBLI2VIG6UZUCHCEMPRJIXVY3RREM45UBPSUQ, seg_pos={0 0}, upload=56512]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:38.549440657 (58h56m9.73973923s) [noise=true , address=79.127.201.215:4351 , piece_id=EYZW7N5D7ASEIQ7HN7FPVMAEX272IAWYQ4JZFKZ4VQLA3CC4PJOA, seg_pos={0 0}, upload=56512]: Done uploading piece. err: <nil> Jul 26 22:31:59.058679761 (58h56m30.248978343s) [noise=true , address=77.90.31.15:4715 , piece_id=FQIE3NHW2XY4GRTQPGJZOWJ6ZXNZ3WZKBL72252X3GVEO7QY2ZVQ, seg_pos={0 0}, upload=56512]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 139.59.144.22:42702->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:31:59.058713600 (58h56m30.249012183s) [noise=true , address=77.90.31.16:4765 , piece_id=OU36US5WMWIWULNGPTNGDXOQSRW7N7CLXCCHJ2YVISL2LHFYNA3Q, seg_pos={0 0}, upload=56512]: Done uploading piece. err: ecclient: upload failed (node:12CSrtv2fMmFBx9zje9zBevpVzxqyhH7fL7VnHbBFEYFRPuwCKK, address:77.90.31.16:4765): protocol: rpc: dial tcp 77.90.31.16:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:31:59.058736403 (58h56m30.249034992s) [seg_pos={0 0}, upload=56512]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:31:59.166581767 (58h56m30.35688035s) [upload=56512]: Done uploading segments... Jul 26 22:31:59.166615586 (58h56m30.356914167s) [upload=56512]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.110703632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X.X..X..XXX....X.X.X...............X.................] duration: 2.292424745s FAIL (2025-07-26 22:31:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.292456682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XX.X..................X.X.........................X......] duration: 2.278448627s FAIL (2025-07-26 22:31:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.278479027s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........................................X.......X..X...X.X] duration: 1m0.001318022s FAIL (2025-07-26 22:31:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00136282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.............................X.............XX...XXXXX.XXXX.X] duration: 1m0.115324867s FAIL (2025-07-26 22:30:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 30.258504456s) [upload=56496]: Got next segment. Inline: false Jul 26 22:29:59.068420498 (58h54m30.258719084s) [upload=56496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:29:59.068431818 (58h54m30.258730407s) [upload=56496]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:29:59.301078056 (58h54m30.491376653s) [seg_pos={0 0}, upload=56496]: Begin upload segment... Jul 26 22:29:59.301197380 (58h54m30.491495963s) [seg_pos={0 0}, upload=56496]: Done begin upload segment. Jul 26 22:29:59.301244572 (58h54m30.491543163s) [noise=true , address=77.90.31.19:4874 , piece_id=KSH2H2VVFN2G45F23T32ER3WYATZEIFVJULHSSBTELNMRMLMGK7Q, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.302015002 (58h54m30.492313593s) [noise=true , address=64.49.13.140:4514 , piece_id=VDZ4QOOXAQUUWMIYLK6FZEOKKJLY2VCYEOE6N6S5EM36ZCSWKPZA, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.302042027 (58h54m30.492340613s) [noise=true , address=77.90.31.11:4573 , piece_id=KXISIGIMNKAARL4BJTGZXQY5DIP3JQ43AA6VQFFNVUXCP3XR7PGA, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.302511684 (58h54m30.492810274s) [noise=true , address=77.90.31.18:4821 , piece_id=QQNFPIGIOBTRHFNCRNV4VWP4JYP7DCZAN6OG5MYLPQENEFDQV6DQ, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.302551846 (58h54m30.492851114s) [noise=true , address=79.127.205.235:4687 , piece_id=MGCB6UIHXDYSAWJ3LUYRZWSHTGDLEH6M5TOA66TLVIKTJIA7454Q, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.302902036 (58h54m30.493200618s) [noise=true , address=79.127.219.51:4210 , piece_id=D2JWUTBMTRX3JG6SE7TVZDJRM2YJWV7C6XE47LE55IGBARKZPVVQ, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.303099412 (58h54m30.493398001s) [noise=true , address=156.146.43.225:4448 , piece_id=BFXYCBLUV44WHUM2RQJVUM3UHCDS6VWT4DJCED362ECSVY3TKG7A, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.303208236 (58h54m30.493506815s) [noise=true , address=64.49.13.169:4716 , piece_id=YGXLY6AKATDSIVLQGVNCEUG2I7GGJ3J42DMET7XTWLFUNGYQF6AQ, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.303541886 (58h54m30.493840463s) [noise=true , address=79.127.205.234:4618 , piece_id=RWRBELEOCB4GDL7U74YSDINSF5MEQ3I7AS2QWUO52ZPQN6FB3E5Q, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.303657137 (58h54m30.493955726s) [noise=true , address=79.127.205.233:4634 , piece_id=B7CXFO46BSSMHENOXFSL47LEREHT7DOLI2UO7GTCGVXDAKOIUWEA, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.303920678 (58h54m30.494219277s) [noise=true , address=79.127.201.215:4349 , piece_id=ULB37RIKK23YJM7Y5KPCKIRWNG6QNUHSUEVUDSFY7LMIPSPL73AA, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.304196960 (58h54m30.494495551s) [noise=true , address=45.91.116.14:4206 , piece_id=6PZ3TTZQMSHCTF2SHZNDF43N5MFKHYKQY26CXYPD7AUZTUQFNBDQ, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.304443778 (58h54m30.494742362s) [noise=true , address=193.149.164.143:4702 , piece_id=PSQJDJ7R2ZYGSPCHH6JHGG7GZ3P3OOD7FHWNGKYXTTAC2TAZP6YA, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.304748476 (58h54m30.495047067s) [noise=true , address=64.49.13.138:4482 , piece_id=SO7EYYZJFKAHRGWYAS3TGFMPNLERALDFUKQ7KTVBPXKDGKQMP4CQ, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.304765739 (58h54m30.495064318s) [noise=true , address=77.90.31.12:4605 , piece_id=DDEB2LH3E2YDNSQM4YAJQO77RFBJS2MTSTYVASUYMVLTLQH6GV5Q, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.305070442 (58h54m30.495369038s) [noise=true , address=79.127.220.101:4048 , piece_id=7OUBPIC2D3EOPA42AZZTAJW56Z7QPEZIUHLFL4T6QB7534U6ICDQ, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.305186555 (58h54m30.495485142s) [noise=true , address=79.127.219.38:4013 , piece_id=HP2ZVMQVBOKLLUKQZJMW33ZXYQVII3HZTEIAUVS2CLD4FSVXYWBQ, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.305458640 (58h54m30.495757231s) [noise=true , address=45.91.116.5:4310 , piece_id=P2M2K7NCJV7XPJZ56GVCWJFVMTQPYK235JWDOYIH3CAUX27OUDHQ, seg_pos={0 0}, upload=56496]: Uploading piece... Jul 26 22:29:59.734563356 (58h54m30.924861938s) [noise=true , address=77.90.31.19:4874 , piece_id=KSH2H2VVFN2G45F23T32ER3WYATZEIFVJULHSSBTELNMRMLMGK7Q, seg_pos={0 0}, upload=56496]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: storage node overloaded, request limit: 2000 Jul 26 22:29:59.861387643 (58h54m31.051686222s) [upload=56496]: Next returned nil segment Jul 26 22:29:59.861408633 (58h54m31.051707221s) [upload=56496]: Waiting for error group managing segments... Jul 26 22:29:59.895094106 (58h54m31.085392687s) [noise=true , address=156.146.43.225:4448 , piece_id=BFXYCBLUV44WHUM2RQJVUM3UHCDS6VWT4DJCED362ECSVY3TKG7A, seg_pos={0 0}, upload=56496]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:00.039602869 (58h54m31.229901439s) [noise=true , address=77.90.31.11:4573 , piece_id=KXISIGIMNKAARL4BJTGZXQY5DIP3JQ43AA6VQFFNVUXCP3XR7PGA, seg_pos={0 0}, upload=56496]: Done uploading piece. err: <nil> Jul 26 22:30:00.050735158 (58h54m31.241033738s) [noise=true , address=77.90.31.12:4605 , piece_id=DDEB2LH3E2YDNSQM4YAJQO77RFBJS2MTSTYVASUYMVLTLQH6GV5Q, seg_pos={0 0}, upload=56496]: Done uploading piece. err: <nil> Jul 26 22:30:00.075168491 (58h54m31.265467075s) [noise=true , address=79.127.219.38:4013 , piece_id=HP2ZVMQVBOKLLUKQZJMW33ZXYQVII3HZTEIAUVS2CLD4FSVXYWBQ, seg_pos={0 0}, upload=56496]: Done uploading piece. err: <nil> Jul 26 22:30:00.101508287 (58h54m31.291806876s) [noise=true , address=64.49.13.138:4482 , piece_id=SO7EYYZJFKAHRGWYAS3TGFMPNLERALDFUKQ7KTVBPXKDGKQMP4CQ, seg_pos={0 0}, upload=56496]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:00.247304548 (58h54m31.43760314s) [noise=true , address=64.49.13.169:4716 , piece_id=YGXLY6AKATDSIVLQGVNCEUG2I7GGJ3J42DMET7XTWLFUNGYQF6AQ, seg_pos={0 0}, upload=56496]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:00.551215498 (58h54m31.741514092s) [noise=true , address=64.49.13.140:4514 , piece_id=VDZ4QOOXAQUUWMIYLK6FZEOKKJLY2VCYEOE6N6S5EM36ZCSWKPZA, seg_pos={0 0}, upload=56496]: Done uploading piece. err: <nil> Jul 26 22:30:00.567594958 (58h54m31.757893548s) [noise=true , address=79.127.205.235:4687 , piece_id=MGCB6UIHXDYSAWJ3LUYRZWSHTGDLEH6M5TOA66TLVIKTJIA7454Q, seg_pos={0 0}, upload=56496]: Done uploading piece. err: <nil> Jul 26 22:30:00.598043177 (58h54m31.788341766s) [noise=true , address=79.127.205.233:4634 , piece_id=B7CXFO46BSSMHENOXFSL47LEREHT7DOLI2UO7GTCGVXDAKOIUWEA, seg_pos={0 0}, upload=56496]: Done uploading piece. err: <nil> Jul 26 22:30:00.605747702 (58h54m31.796046274s) [noise=true , address=79.127.205.234:4618 , piece_id=RWRBELEOCB4GDL7U74YSDINSF5MEQ3I7AS2QWUO52ZPQN6FB3E5Q, seg_pos={0 0}, upload=56496]: Done uploading piece. err: <nil> Jul 26 22:30:00.906768147 (58h54m32.097066738s) [noise=true , address=45.91.116.14:4206 , piece_id=6PZ3TTZQMSHCTF2SHZNDF43N5MFKHYKQY26CXYPD7AUZTUQFNBDQ, seg_pos={0 0}, upload=56496]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:01.116768470 (58h54m32.30706705s) [noise=true , address=79.127.219.51:4210 , piece_id=D2JWUTBMTRX3JG6SE7TVZDJRM2YJWV7C6XE47LE55IGBARKZPVVQ, seg_pos={0 0}, upload=56496]: Done uploading piece. err: <nil> Jul 26 22:30:01.457530102 (58h54m32.647828679s) [noise=true , address=79.127.220.101:4048 , piece_id=7OUBPIC2D3EOPA42AZZTAJW56Z7QPEZIUHLFL4T6QB7534U6ICDQ, seg_pos={0 0}, upload=56496]: Done uploading piece. err: <nil> Jul 26 22:30:03.442387629 (58h54m34.632686215s) [noise=true , address=45.91.116.5:4310 , piece_id=P2M2K7NCJV7XPJZ56GVCWJFVMTQPYK235JWDOYIH3CAUX27OUDHQ, seg_pos={0 0}, upload=56496]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:03.589134924 (58h54m34.779433512s) [noise=true , address=193.149.164.143:4702 , piece_id=PSQJDJ7R2ZYGSPCHH6JHGG7GZ3P3OOD7FHWNGKYXTTAC2TAZP6YA, seg_pos={0 0}, upload=56496]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:25.751840639 (58h54m56.942139213s) [noise=true , address=79.127.201.215:4349 , piece_id=ULB37RIKK23YJM7Y5KPCKIRWNG6QNUHSUEVUDSFY7LMIPSPL73AA, seg_pos={0 0}, upload=56496]: Done uploading piece. err: <nil> Jul 26 22:30:58.789349122 (58h55m29.979647701s) [noise=true , address=77.90.31.18:4821 , piece_id=QQNFPIGIOBTRHFNCRNV4VWP4JYP7DCZAN6OG5MYLPQENEFDQV6DQ, seg_pos={0 0}, upload=56496]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:30:58.791116405 (58h55m29.981414978s) [seg_pos={0 0}, upload=56496]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:30:58.902686934 (58h55m30.0929855s) [upload=56496]: Done uploading segments... Jul 26 22:30:58.902713507 (58h55m30.093012086s) [upload=56496]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.115358198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 982.281553ms FAIL (2025-07-26 22:30:54 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 982.316152ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XX.X..................X.X.........................X.....] duration: 2.254905197s FAIL (2025-07-26 22:30:47 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.254936823s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........X.X..X..XXX....X.X.X...............X................] duration: 2.376185142s FAIL (2025-07-26 22:30:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.376213653s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 808.288041ms FAIL (2025-07-26 22:30:06 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 808.324895ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X.......X..X...X.] duration: 19.496149588s FAIL (2025-07-26 22:29:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.496171709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XX.X..................X.X.........................X....] duration: 2.357405311s FAIL (2025-07-26 22:29:38 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.357442225s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 853.039203ms FAIL (2025-07-26 22:29:36 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 853.074833ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........X.X..X..XXX....X.X.X...............X...............] duration: 2.591398068s FAIL (2025-07-26 22:29:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.591424918s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [..............................X.............XX...XXXXX.XXXX.] duration: 26.748698932s FAIL (2025-07-26 22:29:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== IXZA5SDXALBPDYM4EJ3MPY6OQODZ7NVBSV5OEBWXKZHQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.001332733 (58h52m35.191631339s) [noise=true , address=193.149.164.143:4702 , piece_id=7MWRI3GMMACHPDWEQTRYOGPHWDLXDQPF42D5DQMNUWDXFKT5LQQA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.001511295 (58h52m35.191809908s) [noise=true , address=79.127.205.234:4612 , piece_id=24LSA5Q5ZXMK7YUHE7PMIRXKFVAEL4SL5C4XQAIV6Q6B2Q6F5RQA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.002093088 (58h52m35.192391694s) [noise=true , address=104.36.82.55:4228 , piece_id=OWOPLPI6VQKHUPSZJHWIKZLB4NL3GCFZCADD3UTOWJJAYRMRE43Q, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.002150561 (58h52m35.192449164s) [noise=true , address=64.49.13.138:4482 , piece_id=5VX4UFRK3BJZEHUKCC4CUIQNFXCJLVHCQH3NLOSS6QY3QHWWSHGA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.002233534 (58h52m35.192532126s) [noise=true , address=104.36.82.42:4803 , piece_id=YQSSU4MIAXXCR7BLCPAVYY2DEI2PY3E2DNMFLOBPPTHMFMUL7RFQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.002704229 (58h52m35.193002867s) [noise=true , address=104.36.82.13:4204 , piece_id=IKA5F4JUFPR7SDA7ZTZF6B2N3UGAUTHV2A5MGO2YU43SMYN3FMSQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.002803892 (58h52m35.193102493s) [noise=true , address=104.36.82.12:4157 , piece_id=EOI4IZ3BWWPQPJUTNUQ2CUDSR4NGMSX7H65NKJ4CUMOWDDCTPT3Q, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.003382888 (58h52m35.193681501s) [noise=true , address=104.36.82.11:4125 , piece_id=ZCCLTBG2QRH576FRZVQ6FKQVUTVGG2Y3JUS7FMXIJWDZDYOKA2WA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.004252906 (58h52m35.19455149s) [noise=true , address=104.36.82.56:4238 , piece_id=RJIALPTAM6EVJODOGL5NLLN2QI5CRMUX5UCSD3SNA6F673NVYBFA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.005065894 (58h52m35.195364484s) [noise=true , address=79.127.219.51:4201 , piece_id=UR2V66J4QUR4XUUZML7L4KGFA4EDGGWI3J6VOZQJ2YGTGFUMVDMA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.006449532 (58h52m35.196748161s) [noise=true , address=77.90.31.21:4907 , piece_id=3XVQV76TWOCJBJJJBGRYLANRDK6463PWA4MHJGOUZBPDCUQM4LRA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.006914700 (58h52m35.197213304s) [noise=true , address=79.127.205.235:4681 , piece_id=P3K2Q7Z5NB3WR4FQ4TGJABL4WQGDP2CDQR2M3YBYVX524M5HUWTQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.007423070 (58h52m35.197721661s) [noise=true , address=104.36.82.13:4203 , piece_id=5N27VBSRM2A4MXS54SMOLH5XT5E2ZP256JCQVWYG5KA5C7CXUACQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.007865041 (58h52m35.198163614s) [noise=true , address=79.127.201.215:4352 , piece_id=UZKZ72ZGTQPG5KD2JAVLUQFFOVFYWMARTT56C3Q4QVLPVV654FIQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.009553431 (58h52m35.199852022s) [noise=true , address=45.91.116.10:4109 , piece_id=ZSWQZMIAFHFIXJKUILFAFTXTQIVJYMHHUEY4XZIFHJ6CMAENGZKQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.010041215 (58h52m35.200339795s) [noise=true , address=104.36.82.34:4566 , piece_id=M5DXZIKZHD7GJNZD7CC6R5WFTNBP5CO4MFCS2VPGZWU7PG7RR2IA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.015810039 (58h52m35.206108618s) [noise=true , address=104.36.82.40:4973 , piece_id=72LWO6JFEHDC2LTH3U3ZXBZUZ4E55PJRTI4FJPO5B6E4FPFE5GKA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.603744508 (58h52m35.794043075s) [upload=56467]: Next returned nil segment Jul 26 22:28:04.603763605 (58h52m35.794062185s) [upload=56467]: Waiting for error group managing segments... Jul 26 22:28:04.611264705 (58h52m35.801563344s) [noise=true , address=79.127.220.101:4048 , piece_id=UHP2O4U5IXZA5SDXALBPDYM4EJ3MPY6OQODZ7NVBSV5OEBWXKZHQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:28:04.991389376 (58h52m36.18168795s) [noise=true , address=104.36.82.55:4228 , piece_id=OWOPLPI6VQKHUPSZJHWIKZLB4NL3GCFZCADD3UTOWJJAYRMRE43Q, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:05.368755352 (58h52m36.559053919s) [noise=true , address=79.127.205.235:4681 , piece_id=P3K2Q7Z5NB3WR4FQ4TGJABL4WQGDP2CDQR2M3YBYVX524M5HUWTQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:05.401467448 (58h52m36.59176603s) [noise=true , address=79.127.205.234:4612 , piece_id=24LSA5Q5ZXMK7YUHE7PMIRXKFVAEL4SL5C4XQAIV6Q6B2Q6F5RQA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:06.163680219 (58h52m37.353978791s) [noise=true , address=193.149.164.143:4702 , piece_id=7MWRI3GMMACHPDWEQTRYOGPHWDLXDQPF42D5DQMNUWDXFKT5LQQA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:06.185486930 (58h52m37.375785502s) [noise=true , address=79.127.219.51:4201 , piece_id=UR2V66J4QUR4XUUZML7L4KGFA4EDGGWI3J6VOZQJ2YGTGFUMVDMA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:06.332030092 (58h52m37.522328672s) [noise=true , address=64.49.13.138:4482 , piece_id=5VX4UFRK3BJZEHUKCC4CUIQNFXCJLVHCQH3NLOSS6QY3QHWWSHGA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:28:08.468927109 (58h52m39.659225679s) [noise=true , address=45.91.116.10:4109 , piece_id=ZSWQZMIAFHFIXJKUILFAFTXTQIVJYMHHUEY4XZIFHJ6CMAENGZKQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:23.048865492 (58h52m54.239164058s) [noise=true , address=79.127.201.215:4352 , piece_id=UZKZ72ZGTQPG5KD2JAVLUQFFOVFYWMARTT56C3Q4QVLPVV654FIQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:33.486612935 (58h53m4.676911507s) [noise=true , address=104.36.82.56:4238 , piece_id=RJIALPTAM6EVJODOGL5NLLN2QI5CRMUX5UCSD3SNA6F673NVYBFA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:35.451020377 (58h53m6.641318946s) [noise=true , address=77.90.31.21:4907 , piece_id=3XVQV76TWOCJBJJJBGRYLANRDK6463PWA4MHJGOUZBPDCUQM4LRA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:48.631996196 (58h53m19.82229477s) [noise=true , address=104.36.82.13:4203 , piece_id=5N27VBSRM2A4MXS54SMOLH5XT5E2ZP256JCQVWYG5KA5C7CXUACQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:59.830784978 (58h53m31.02108355s) [noise=true , address=104.36.82.12:4157 , piece_id=EOI4IZ3BWWPQPJUTNUQ2CUDSR4NGMSX7H65NKJ4CUMOWDDCTPT3Q, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:29:03.063258011 (58h53m34.253556583s) [noise=true , address=104.36.82.13:4204 , piece_id=IKA5F4JUFPR7SDA7ZTZF6B2N3UGAUTHV2A5MGO2YU43SMYN3FMSQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:29:03.472053584 (58h53m34.662352164s) [noise=true , address=104.36.82.42:4803 , piece_id=YQSSU4MIAXXCR7BLCPAVYY2DEI2PY3E2DNMFLOBPPTHMFMUL7RFQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: ecclient: upload failed (node:12GFk3GkhB88UKQdZM8vZ9VLNuuYDrS5ThZjVAdbL2CtTQVMDEB, address:104.36.82.42:4803): protocol: write tcp 139.59.144.22:36618->104.36.82.42:4803: use of closed network connection; write tcp 139.59.144.22:36618->104.36.82.42:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36618->104.36.82.42:4803: use of closed network connection Jul 26 22:29:03.472114197 (58h53m34.662412775s) [noise=true , address=104.36.82.40:4973 , piece_id=72LWO6JFEHDC2LTH3U3ZXBZUZ4E55PJRTI4FJPO5B6E4FPFE5GKA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: ecclient: upload failed (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx, address:104.36.82.40:4973): protocol: write tcp 139.59.144.22:57760->104.36.82.40:4973: use of closed network connection; write tcp 139.59.144.22:57760->104.36.82.40:4973: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57760->104.36.82.40:4973: use of closed network connection Jul 26 22:29:03.472147182 (58h53m34.662445762s) [noise=true , address=104.36.82.34:4566 , piece_id=M5DXZIKZHD7GJNZD7CC6R5WFTNBP5CO4MFCS2VPGZWU7PG7RR2IA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: ecclient: upload failed (node:1Z9ic1vvfDViKVZbrkLZnryBxKKQ4rJdjBFY6wEYCDtJt9ci77, address:104.36.82.34:4566): protocol: write tcp 139.59.144.22:44242->104.36.82.34:4566: use of closed network connection; write tcp 139.59.144.22:44242->104.36.82.34:4566: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44242->104.36.82.34:4566: use of closed network connection Jul 26 22:29:03.472186977 (58h53m34.662485556s) [noise=true , address=104.36.82.11:4125 , piece_id=ZCCLTBG2QRH576FRZVQ6FKQVUTVGG2Y3JUS7FMXIJWDZDYOKA2WA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:4125): protocol: write tcp 139.59.144.22:56246->104.36.82.11:4125: use of closed network connection; write tcp 139.59.144.22:56246->104.36.82.11:4125: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56246->104.36.82.11:4125: use of closed network connection Jul 26 22:29:03.472298329 (58h53m34.662596896s) [seg_pos={0 0}, upload=56467]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:29:03.574366837 (58h53m34.764665412s) [upload=56467]: Done uploading segments... Jul 26 22:29:03.574398161 (58h53m34.764696743s) [upload=56467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 26.748723265s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............................................X.......X..X...X] duration: 1m0.001158856s FAIL (2025-07-26 22:29:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001187419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [......XX.X..................X.X.........................X...] duration: 2.255364703s FAIL (2025-07-26 22:29:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.255388112s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [...............................X.............XX...XXXXX.XXXX] duration: 1m0.103861872s FAIL (2025-07-26 22:29:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== IXZA5SDXALBPDYM4EJ3MPY6OQODZ7NVBSV5OEBWXKZHQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.001332733 (58h52m35.191631339s) [noise=true , address=193.149.164.143:4702 , piece_id=7MWRI3GMMACHPDWEQTRYOGPHWDLXDQPF42D5DQMNUWDXFKT5LQQA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.001511295 (58h52m35.191809908s) [noise=true , address=79.127.205.234:4612 , piece_id=24LSA5Q5ZXMK7YUHE7PMIRXKFVAEL4SL5C4XQAIV6Q6B2Q6F5RQA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.002093088 (58h52m35.192391694s) [noise=true , address=104.36.82.55:4228 , piece_id=OWOPLPI6VQKHUPSZJHWIKZLB4NL3GCFZCADD3UTOWJJAYRMRE43Q, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.002150561 (58h52m35.192449164s) [noise=true , address=64.49.13.138:4482 , piece_id=5VX4UFRK3BJZEHUKCC4CUIQNFXCJLVHCQH3NLOSS6QY3QHWWSHGA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.002233534 (58h52m35.192532126s) [noise=true , address=104.36.82.42:4803 , piece_id=YQSSU4MIAXXCR7BLCPAVYY2DEI2PY3E2DNMFLOBPPTHMFMUL7RFQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.002704229 (58h52m35.193002867s) [noise=true , address=104.36.82.13:4204 , piece_id=IKA5F4JUFPR7SDA7ZTZF6B2N3UGAUTHV2A5MGO2YU43SMYN3FMSQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.002803892 (58h52m35.193102493s) [noise=true , address=104.36.82.12:4157 , piece_id=EOI4IZ3BWWPQPJUTNUQ2CUDSR4NGMSX7H65NKJ4CUMOWDDCTPT3Q, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.003382888 (58h52m35.193681501s) [noise=true , address=104.36.82.11:4125 , piece_id=ZCCLTBG2QRH576FRZVQ6FKQVUTVGG2Y3JUS7FMXIJWDZDYOKA2WA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.004252906 (58h52m35.19455149s) [noise=true , address=104.36.82.56:4238 , piece_id=RJIALPTAM6EVJODOGL5NLLN2QI5CRMUX5UCSD3SNA6F673NVYBFA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.005065894 (58h52m35.195364484s) [noise=true , address=79.127.219.51:4201 , piece_id=UR2V66J4QUR4XUUZML7L4KGFA4EDGGWI3J6VOZQJ2YGTGFUMVDMA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.006449532 (58h52m35.196748161s) [noise=true , address=77.90.31.21:4907 , piece_id=3XVQV76TWOCJBJJJBGRYLANRDK6463PWA4MHJGOUZBPDCUQM4LRA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.006914700 (58h52m35.197213304s) [noise=true , address=79.127.205.235:4681 , piece_id=P3K2Q7Z5NB3WR4FQ4TGJABL4WQGDP2CDQR2M3YBYVX524M5HUWTQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.007423070 (58h52m35.197721661s) [noise=true , address=104.36.82.13:4203 , piece_id=5N27VBSRM2A4MXS54SMOLH5XT5E2ZP256JCQVWYG5KA5C7CXUACQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.007865041 (58h52m35.198163614s) [noise=true , address=79.127.201.215:4352 , piece_id=UZKZ72ZGTQPG5KD2JAVLUQFFOVFYWMARTT56C3Q4QVLPVV654FIQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.009553431 (58h52m35.199852022s) [noise=true , address=45.91.116.10:4109 , piece_id=ZSWQZMIAFHFIXJKUILFAFTXTQIVJYMHHUEY4XZIFHJ6CMAENGZKQ, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.010041215 (58h52m35.200339795s) [noise=true , address=104.36.82.34:4566 , piece_id=M5DXZIKZHD7GJNZD7CC6R5WFTNBP5CO4MFCS2VPGZWU7PG7RR2IA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.015810039 (58h52m35.206108618s) [noise=true , address=104.36.82.40:4973 , piece_id=72LWO6JFEHDC2LTH3U3ZXBZUZ4E55PJRTI4FJPO5B6E4FPFE5GKA, seg_pos={0 0}, upload=56467]: Uploading piece... Jul 26 22:28:04.603744508 (58h52m35.794043075s) [upload=56467]: Next returned nil segment Jul 26 22:28:04.603763605 (58h52m35.794062185s) [upload=56467]: Waiting for error group managing segments... Jul 26 22:28:04.611264705 (58h52m35.801563344s) [noise=true , address=79.127.220.101:4048 , piece_id=UHP2O4U5IXZA5SDXALBPDYM4EJ3MPY6OQODZ7NVBSV5OEBWXKZHQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:28:04.991389376 (58h52m36.18168795s) [noise=true , address=104.36.82.55:4228 , piece_id=OWOPLPI6VQKHUPSZJHWIKZLB4NL3GCFZCADD3UTOWJJAYRMRE43Q, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:05.368755352 (58h52m36.559053919s) [noise=true , address=79.127.205.235:4681 , piece_id=P3K2Q7Z5NB3WR4FQ4TGJABL4WQGDP2CDQR2M3YBYVX524M5HUWTQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:05.401467448 (58h52m36.59176603s) [noise=true , address=79.127.205.234:4612 , piece_id=24LSA5Q5ZXMK7YUHE7PMIRXKFVAEL4SL5C4XQAIV6Q6B2Q6F5RQA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:06.163680219 (58h52m37.353978791s) [noise=true , address=193.149.164.143:4702 , piece_id=7MWRI3GMMACHPDWEQTRYOGPHWDLXDQPF42D5DQMNUWDXFKT5LQQA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:06.185486930 (58h52m37.375785502s) [noise=true , address=79.127.219.51:4201 , piece_id=UR2V66J4QUR4XUUZML7L4KGFA4EDGGWI3J6VOZQJ2YGTGFUMVDMA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:06.332030092 (58h52m37.522328672s) [noise=true , address=64.49.13.138:4482 , piece_id=5VX4UFRK3BJZEHUKCC4CUIQNFXCJLVHCQH3NLOSS6QY3QHWWSHGA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:28:08.468927109 (58h52m39.659225679s) [noise=true , address=45.91.116.10:4109 , piece_id=ZSWQZMIAFHFIXJKUILFAFTXTQIVJYMHHUEY4XZIFHJ6CMAENGZKQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:23.048865492 (58h52m54.239164058s) [noise=true , address=79.127.201.215:4352 , piece_id=UZKZ72ZGTQPG5KD2JAVLUQFFOVFYWMARTT56C3Q4QVLPVV654FIQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:33.486612935 (58h53m4.676911507s) [noise=true , address=104.36.82.56:4238 , piece_id=RJIALPTAM6EVJODOGL5NLLN2QI5CRMUX5UCSD3SNA6F673NVYBFA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:35.451020377 (58h53m6.641318946s) [noise=true , address=77.90.31.21:4907 , piece_id=3XVQV76TWOCJBJJJBGRYLANRDK6463PWA4MHJGOUZBPDCUQM4LRA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:48.631996196 (58h53m19.82229477s) [noise=true , address=104.36.82.13:4203 , piece_id=5N27VBSRM2A4MXS54SMOLH5XT5E2ZP256JCQVWYG5KA5C7CXUACQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:28:59.830784978 (58h53m31.02108355s) [noise=true , address=104.36.82.12:4157 , piece_id=EOI4IZ3BWWPQPJUTNUQ2CUDSR4NGMSX7H65NKJ4CUMOWDDCTPT3Q, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:29:03.063258011 (58h53m34.253556583s) [noise=true , address=104.36.82.13:4204 , piece_id=IKA5F4JUFPR7SDA7ZTZF6B2N3UGAUTHV2A5MGO2YU43SMYN3FMSQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: <nil> Jul 26 22:29:03.472053584 (58h53m34.662352164s) [noise=true , address=104.36.82.42:4803 , piece_id=YQSSU4MIAXXCR7BLCPAVYY2DEI2PY3E2DNMFLOBPPTHMFMUL7RFQ, seg_pos={0 0}, upload=56467]: Done uploading piece. err: ecclient: upload failed (node:12GFk3GkhB88UKQdZM8vZ9VLNuuYDrS5ThZjVAdbL2CtTQVMDEB, address:104.36.82.42:4803): protocol: write tcp 139.59.144.22:36618->104.36.82.42:4803: use of closed network connection; write tcp 139.59.144.22:36618->104.36.82.42:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36618->104.36.82.42:4803: use of closed network connection Jul 26 22:29:03.472114197 (58h53m34.662412775s) [noise=true , address=104.36.82.40:4973 , piece_id=72LWO6JFEHDC2LTH3U3ZXBZUZ4E55PJRTI4FJPO5B6E4FPFE5GKA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: ecclient: upload failed (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx, address:104.36.82.40:4973): protocol: write tcp 139.59.144.22:57760->104.36.82.40:4973: use of closed network connection; write tcp 139.59.144.22:57760->104.36.82.40:4973: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57760->104.36.82.40:4973: use of closed network connection Jul 26 22:29:03.472147182 (58h53m34.662445762s) [noise=true , address=104.36.82.34:4566 , piece_id=M5DXZIKZHD7GJNZD7CC6R5WFTNBP5CO4MFCS2VPGZWU7PG7RR2IA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: ecclient: upload failed (node:1Z9ic1vvfDViKVZbrkLZnryBxKKQ4rJdjBFY6wEYCDtJt9ci77, address:104.36.82.34:4566): protocol: write tcp 139.59.144.22:44242->104.36.82.34:4566: use of closed network connection; write tcp 139.59.144.22:44242->104.36.82.34:4566: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44242->104.36.82.34:4566: use of closed network connection Jul 26 22:29:03.472186977 (58h53m34.662485556s) [noise=true , address=104.36.82.11:4125 , piece_id=ZCCLTBG2QRH576FRZVQ6FKQVUTVGG2Y3JUS7FMXIJWDZDYOKA2WA, seg_pos={0 0}, upload=56467]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:4125): protocol: write tcp 139.59.144.22:56246->104.36.82.11:4125: use of closed network connection; write tcp 139.59.144.22:56246->104.36.82.11:4125: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56246->104.36.82.11:4125: use of closed network connection Jul 26 22:29:03.472298329 (58h53m34.662596896s) [seg_pos={0 0}, upload=56467]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:29:03.574366837 (58h53m34.764665412s) [upload=56467]: Done uploading segments... Jul 26 22:29:03.574398161 (58h53m34.764696743s) [upload=56467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.103899154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........X.X..X..XXX....X.X.X...............X..............] duration: 2.286111427s FAIL (2025-07-26 22:28:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.286146429s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......XX.X..................X.X.........................X..] duration: 2.377770522s FAIL (2025-07-26 22:28:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.377876102s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 895.483458ms FAIL (2025-07-26 22:28:45 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 895.525605ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [........XX.X..................X.X.........................X.] duration: 3.36223054s FAIL (2025-07-26 22:28:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.362286991s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [................................X.............XX...XXXXX.XXX] duration: 1m0.124549231s FAIL (2025-07-26 22:28:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8]: Starting upload Jul 26 22:27:03.503921040 (58h51m34.694219631s) [upload=56448]: Done starting upload Jul 26 22:27:03.504010442 (58h51m34.694309015s) [upload=56448]: Uploading segments... Jul 26 22:27:03.504343632 (58h51m34.69464222s) [upload=56448]: Got next segment. Inline: false Jul 26 22:27:03.504395541 (58h51m34.69469412s) [upload=56448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:27:03.504405266 (58h51m34.694703845s) [upload=56448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:27:03.704553672 (58h51m34.894852264s) [seg_pos={0 0}, upload=56448]: Begin upload segment... Jul 26 22:27:03.704643332 (58h51m34.894941922s) [seg_pos={0 0}, upload=56448]: Done begin upload segment. Jul 26 22:27:03.704697157 (58h51m34.894995981s) [noise=true , address=104.36.82.52:4086 , piece_id=HA4UCLWN4GMI7ETKKO3KNZNIDFQ6BZ5TKGMH5TEIXUVTDKTZQEAA, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.704707544 (58h51m34.895006128s) [noise=true , address=79.127.201.215:4344 , piece_id=OGW3BU63CJVNKWUR5DVMG5GG546A4O5J74G6QTWWIIWBBRP4M3UQ, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.705190183 (58h51m34.895488775s) [noise=true , address=104.36.82.10:4098 , piece_id=MNFZJZRFSOD22QJXZ3QSBIBLHB4367PV5NSYTGZN4UVZQZFYQU4A, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.705249235 (58h51m34.895547827s) [noise=true , address=104.36.82.16:4300 , piece_id=EGQ3BVGKOFEPQWXQ6OUQKB6UPXCZHVZV5J4KQI5XS464MHABT34Q, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.705575128 (58h51m34.895873718s) [noise=true , address=104.36.82.35:4603 , piece_id=A7DLP6MCT5GIY4XEIU3L6DERF3IG32VYGJZEYZ36U2C4PQ6FWI2A, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.705750121 (58h51m34.896048712s) [noise=true , address=79.127.220.101:4048 , piece_id=UMNAG3ZOPQ7SDGXY252VRAQ7OI2UANQCLTUEYMEYDREMTJR5U2PA, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.705889712 (58h51m34.896188295s) [noise=true , address=104.36.82.13:4193 , piece_id=2SGUMUPEMRQRK76BT23VB34WMXMZ2EMA67VVB3RMICQYHUONPRRQ, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.706215191 (58h51m34.896513775s) [noise=true , address=104.36.82.19:4401 , piece_id=ZN4YZLCSJMXNYD3LVYHY33F5YFUDVV3M3WQZDLEJBV4R42CDGB7A, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.706373893 (58h51m34.896672498s) [noise=true , address=79.127.205.245:4221 , piece_id=3BNOD43OYOOP65NBQAXGVN3Y7ZR3TXHOUKFUYXRID4FRBJQLQGCA, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.706549783 (58h51m34.89684837s) [noise=true , address=156.146.43.225:4434 , piece_id=VFHBKKFDOPIU2IX77YEX4CV6ADV7VNJBJGRS2PFVR5VFQWZWKJWA, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.706883972 (58h51m34.897182554s) [noise=true , address=104.36.82.7:4920 , piece_id=C46IGCW3EEIKDLBMJCKTHC6TFGYAKLR4HSERNQXFGD4US4XW2MLQ, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.706921591 (58h51m34.897220181s) [noise=true , address=77.90.31.11:4560 , piece_id=4P3VQLMJGWZ5I6HMRG3DUX32TMHG265VTFKS7PKRFWMUERP5GB6Q, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.707259391 (58h51m34.897557977s) [noise=true , address=104.36.82.56:4232 , piece_id=VT5B54SAZFGVYV2SUQPMREDTNY2AZ6SSILFLM7LEM4RSXTDR7GJA, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.707343784 (58h51m34.897642375s) [noise=true , address=104.36.82.21:4486 , piece_id=SDDID2DD47FHW6SJIOZUZBEGMWF7IGAMINSZWH6SXYPGNYW63I2Q, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.707560125 (58h51m34.897858709s) [noise=true , address=104.36.82.52:4087 , piece_id=N7DECNGPRCNMMFTIVSJEFFGPEEWMGBY6TUBQJY2B5XVHARYFFA5A, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.707864905 (58h51m34.898163498s) [noise=true , address=104.36.82.4:4823 , piece_id=K2AMOR5JCRTDM4EWI47HGOAM7IFDTCFZXH5AMW5MTDOHRRUG4AXA, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.708338593 (58h51m34.898637183s) [noise=true , address=79.127.205.235:4669 , piece_id=EEYLHNTAXJ3LNLFVDMHSIA6XUF3DWOCA6MPPQZNZ5VDMGQSUPOJA, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:03.708781231 (58h51m34.899079819s) [noise=true , address=79.127.205.233:4652 , piece_id=IEU3M5DEA6ETDCH324HZ2H6OFSYUL5HJYEPHBQFZ3T7GXGFJX7CQ, seg_pos={0 0}, upload=56448]: Uploading piece... Jul 26 22:27:04.290286458 (58h51m35.480585025s) [upload=56448]: Next returned nil segment Jul 26 22:27:04.290311407 (58h51m35.480610002s) [upload=56448]: Waiting for error group managing segments... Jul 26 22:27:04.481383232 (58h51m35.671681804s) [noise=true , address=77.90.31.11:4560 , piece_id=4P3VQLMJGWZ5I6HMRG3DUX32TMHG265VTFKS7PKRFWMUERP5GB6Q, seg_pos={0 0}, upload=56448]: Done uploading piece. err: <nil> Jul 26 22:27:05.020406499 (58h51m36.210705073s) [noise=true , address=79.127.205.245:4221 , piece_id=3BNOD43OYOOP65NBQAXGVN3Y7ZR3TXHOUKFUYXRID4FRBJQLQGCA, seg_pos={0 0}, upload=56448]: Done uploading piece. err: <nil> Jul 26 22:27:05.042490853 (58h51m36.232789434s) [noise=true , address=79.127.205.233:4652 , piece_id=IEU3M5DEA6ETDCH324HZ2H6OFSYUL5HJYEPHBQFZ3T7GXGFJX7CQ, seg_pos={0 0}, upload=56448]: Done uploading piece. err: <nil> Jul 26 22:27:05.118776286 (58h51m36.309074859s) [noise=true , address=79.127.205.235:4669 , piece_id=EEYLHNTAXJ3LNLFVDMHSIA6XUF3DWOCA6MPPQZNZ5VDMGQSUPOJA, seg_pos={0 0}, upload=56448]: Done uploading piece. err: <nil> Jul 26 22:27:07.905776527 (58h51m39.096075102s) [noise=true , address=79.127.220.101:4048 , piece_id=UMNAG3ZOPQ7SDGXY252VRAQ7OI2UANQCLTUEYMEYDREMTJR5U2PA, seg_pos={0 0}, upload=56448]: Done uploading piece. err: <nil> Jul 26 22:27:08.933211653 (58h51m40.123510218s) [noise=true , address=156.146.43.225:4434 , piece_id=VFHBKKFDOPIU2IX77YEX4CV6ADV7VNJBJGRS2PFVR5VFQWZWKJWA, seg_pos={0 0}, upload=56448]: Done uploading piece. err: <nil> Jul 26 22:27:15.047683451 (58h51m46.237982036s) [noise=true , address=79.127.201.215:4344 , piece_id=OGW3BU63CJVNKWUR5DVMG5GG546A4O5J74G6QTWWIIWBBRP4M3UQ, seg_pos={0 0}, upload=56448]: Done uploading piece. err: <nil> Jul 26 22:27:18.688391950 (58h51m49.878690541s) [noise=true , address=104.36.82.56:4232 , piece_id=VT5B54SAZFGVYV2SUQPMREDTNY2AZ6SSILFLM7LEM4RSXTDR7GJA, seg_pos={0 0}, upload=56448]: Done uploading piece. err: <nil> Jul 26 22:27:20.515521950 (58h51m51.705820516s) [noise=true , address=104.36.82.52:4086 , piece_id=HA4UCLWN4GMI7ETKKO3KNZNIDFQ6BZ5TKGMH5TEIXUVTDKTZQEAA, seg_pos={0 0}, upload=56448]: Done uploading piece. err: <nil> Jul 26 22:27:27.657444107 (58h51m58.847742677s) [noise=true , address=104.36.82.52:4087 , piece_id=N7DECNGPRCNMMFTIVSJEFFGPEEWMGBY6TUBQJY2B5XVHARYFFA5A, seg_pos={0 0}, upload=56448]: Done uploading piece. err: <nil> Jul 26 22:27:48.899104725 (58h52m20.089403291s) [noise=true , address=104.36.82.4:4823 , piece_id=K2AMOR5JCRTDM4EWI47HGOAM7IFDTCFZXH5AMW5MTDOHRRUG4AXA, seg_pos={0 0}, upload=56448]: Done uploading piece. err: <nil> Jul 26 22:27:56.218826000 (58h52m27.40912458s) [noise=true , address=104.36.82.7:4920 , piece_id=C46IGCW3EEIKDLBMJCKTHC6TFGYAKLR4HSERNQXFGD4US4XW2MLQ, seg_pos={0 0}, upload=56448]: Done uploading piece. err: <nil> Jul 26 22:28:02.120989854 (58h52m33.311288426s) [noise=true , address=104.36.82.16:4300 , piece_id=EGQ3BVGKOFEPQWXQ6OUQKB6UPXCZHVZV5J4KQI5XS464MHABT34Q, seg_pos={0 0}, upload=56448]: Done uploading piece. err: <nil> Jul 26 22:28:03.192005807 (58h52m34.382304389s) [noise=true , address=104.36.82.10:4098 , piece_id=MNFZJZRFSOD22QJXZ3QSBIBLHB4367PV5NSYTGZN4UVZQZFYQU4A, seg_pos={0 0}, upload=56448]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:28:03.192057029 (58h52m34.382355602s) [noise=true , address=104.36.82.35:4603 , piece_id=A7DLP6MCT5GIY4XEIU3L6DERF3IG32VYGJZEYZ36U2C4PQ6FWI2A, seg_pos={0 0}, upload=56448]: Done uploading piece. err: ecclient: upload failed (node:12ASAeTMBu4sY1GuZiSuwTXmhPksA1Te2BVZdcoE9QvpLAmcQX3, address:104.36.82.35:4603): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:28:03.192102979 (58h52m34.382401549s) [noise=true , address=104.36.82.13:4193 , piece_id=2SGUMUPEMRQRK76BT23VB34WMXMZ2EMA67VVB3RMICQYHUONPRRQ, seg_pos={0 0}, upload=56448]: Done uploading piece. err: ecclient: upload failed (node:15XNLgptmXprcfkVMJsuzY2WDyHiZDpLetiBpmjXhb4c9q7yMd, address:104.36.82.13:4193): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:28:03.192187636 (58h52m34.382486206s) [noise=true , address=104.36.82.19:4401 , piece_id=ZN4YZLCSJMXNYD3LVYHY33F5YFUDVV3M3WQZDLEJBV4R42CDGB7A, seg_pos={0 0}, upload=56448]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:104.36.82.19:4401): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:28:03.192457411 (58h52m34.382755987s) [noise=true , address=104.36.82.21:4486 , piece_id=SDDID2DD47FHW6SJIOZUZBEGMWF7IGAMINSZWH6SXYPGNYW63I2Q, seg_pos={0 0}, upload=56448]: Done uploading piece. err: ecclient: upload failed (node:12jvSGi6s5yi55BBFJ4aErrap64wGP2WbDL9oUB6pdGiv8DM385, address:104.36.82.21:4486): protocol: write tcp 139.59.144.22:53090->104.36.82.21:4486: use of closed network connection; write tcp 139.59.144.22:53090->104.36.82.21:4486: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53090->104.36.82.21:4486: use of closed network connection Jul 26 22:28:03.192506267 (58h52m34.382804847s) [seg_pos={0 0}, upload=56448]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:28:03.314085128 (58h52m34.504383709s) [upload=56448]: Done uploading segments... Jul 26 22:28:03.314123112 (58h52m34.504421698s) [upload=56448]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.124569581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........XX.X..................X.X.........................X] duration: 30.106480698s FAIL (2025-07-26 22:27:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.106517921s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........X.X..X..XXX....X.X.X...............X.............] duration: 2.426530136s FAIL (2025-07-26 22:27:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.426560954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 926.733412ms FAIL (2025-07-26 22:27:48 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 926.85138ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............X.X..X..XXX....X.X.X...............X............] duration: 2.397888328s FAIL (2025-07-26 22:27:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.397936269s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X.......X..X...] duration: 5.560595958s FAIL (2025-07-26 22:27:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.560634252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.............X.X..X..XXX....X.X.X...............X...........] duration: 2.265195815s FAIL (2025-07-26 22:27:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.26522688s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.590224777s FAIL (2025-07-26 22:27:05 UTC) orbiter warning: context deadline exceeded Duration: 2.590255121s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.................................X.............XX...XXXXX.XX] duration: 1m0.108428705s FAIL (2025-07-26 22:27:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 50m34.418110151s) [upload=56433]: Got next segment. Inline: false Jul 26 22:26:03.227893927 (58h50m34.418192496s) [upload=56433]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:26:03.227917744 (58h50m34.418216324s) [upload=56433]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:26:03.427077208 (58h50m34.617375796s) [seg_pos={0 0}, upload=56433]: Begin upload segment... Jul 26 22:26:03.427132636 (58h50m34.61743122s) [seg_pos={0 0}, upload=56433]: Done begin upload segment. Jul 26 22:26:03.427169421 (58h50m34.617468003s) [noise=true , address=79.127.201.215:4350 , piece_id=SLAQNUXNJJTICL6PWWD6VBNMOCDNODPDCKQSLHESIJITWNNNWZSA, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.482952952 (58h50m34.67325154s) [noise=true , address=77.90.31.18:4821 , piece_id=AYMNMUPFUHZFB6NBKFJIAZJQCJBEIKZB43WUIIAC7HED7XMDD4YQ, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.483516056 (58h50m34.673814639s) [noise=true , address=77.90.31.12:4615 , piece_id=FCFU2LUVQGZ73JKAEFO677L36LQHEXXDKHB2G5NOV5NMZ624FADA, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.483971201 (58h50m34.67426978s) [noise=true , address=104.36.82.7:4921 , piece_id=D7RSTDEGZQQ67IPAW4UIKDOWR2JEDLMYNGI62XYXY2NGNFRYAMOQ, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.484490365 (58h50m34.674788949s) [noise=true , address=104.36.82.52:4087 , piece_id=QNQPJ76ECSGCZAIKVDZPMVNTRGBEZUQC7XA4OL7EQ2NWMIZFSTUQ, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.485124438 (58h50m34.675423022s) [noise=true , address=104.36.82.4:4828 , piece_id=D4A7YOMNKUK2P7EFJ7TWBJMAB4XVNUKXFSAIE522IKTQ6QMB3P3Q, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.485729399 (58h50m34.676027988s) [noise=true , address=156.146.43.225:4434 , piece_id=SF57F4DWPESIJDDRIIELIW6U2LWPRVDDP2ZAQ37PJRNKMJZP3SPQ, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.486433284 (58h50m34.676731874s) [noise=true , address=79.127.205.235:4680 , piece_id=DT7W7FPKQ2UM36C5BOD7EQCIIQD3T3RZIFTYL76CV7BY55MQDAFQ, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.487987025 (58h50m34.678285616s) [noise=true , address=104.36.82.12:4169 , piece_id=XVZ7WJNU6PGADQPHQ36BJANJPIRF4FERZ7ZRWMWTSWK6PGZKYWJQ, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.488541659 (58h50m34.678840241s) [noise=true , address=104.36.82.21:4487 , piece_id=JCXM5YBQNM7POAGPCG4MN4VPIGLTEOGDNBO3RFCIED3Z46HGUL2Q, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.489043060 (58h50m34.679341646s) [noise=true , address=104.36.82.55:4228 , piece_id=FDUWZQFR7KPANQ2Q7BBN35MAVMVPTIRAR43FKSIWWNFSEQEYJ2WQ, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.489589324 (58h50m34.679887903s) [noise=true , address=79.127.219.38:4017 , piece_id=SBGENO5MLPCOZQUQR4AJRFMJRKV2OVDNG4I5EXHCOMRECD4TEKRQ, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.490707957 (58h50m34.681006545s) [noise=true , address=79.127.205.246:4286 , piece_id=LLMIQNN2MC4CVQ3A2IKSCESOZQIKPVTNO7BU6GNGZDK6FYW3HW7A, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.491231611 (58h50m34.681530199s) [noise=true , address=77.90.31.15:4703 , piece_id=DMK6L5MKPGMZWPFYCFERUKXWK27QQOBE3QUMRN6IHHRLCU7HUWHQ, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.491713397 (58h50m34.682011985s) [noise=true , address=79.127.205.234:4610 , piece_id=3P3BMTA7FVBI67YE4GB566UTEBXA7XIIRVVIWDFDT6QDOZGB2APA, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.492195034 (58h50m34.682493629s) [noise=true , address=79.127.205.245:4252 , piece_id=7I4XQOUS4K4DLCN7ABMVYBTMMDLBMJ35N636J3PXIPQEXRPX2Q5A, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.492674866 (58h50m34.682973446s) [noise=true , address=104.36.82.56:4236 , piece_id=UOIFGBYHYQBU3H255M2IBSG337FA2OIO63PLIR2IO5ROCT22245A, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:03.493127973 (58h50m34.683426553s) [noise=true , address=64.49.13.138:4482 , piece_id=OIYY6RWLKYXBIU6Y3V6V7PDC37YCNAWQTB2OWZ4ZEQ4P7YMC6OQQ, seg_pos={0 0}, upload=56433]: Uploading piece... Jul 26 22:26:04.127623861 (58h50m35.317922439s) [upload=56433]: Next returned nil segment Jul 26 22:26:04.127649912 (58h50m35.317948501s) [upload=56433]: Waiting for error group managing segments... Jul 26 22:26:04.352760982 (58h50m35.543059564s) [noise=true , address=77.90.31.15:4703 , piece_id=DMK6L5MKPGMZWPFYCFERUKXWK27QQOBE3QUMRN6IHHRLCU7HUWHQ, seg_pos={0 0}, upload=56433]: Done uploading piece. err: <nil> Jul 26 22:26:04.557581056 (58h50m35.747879628s) [noise=true , address=79.127.219.38:4017 , piece_id=SBGENO5MLPCOZQUQR4AJRFMJRKV2OVDNG4I5EXHCOMRECD4TEKRQ, seg_pos={0 0}, upload=56433]: Done uploading piece. err: <nil> Jul 26 22:26:04.789402189 (58h50m35.979700769s) [noise=true , address=79.127.205.234:4610 , piece_id=3P3BMTA7FVBI67YE4GB566UTEBXA7XIIRVVIWDFDT6QDOZGB2APA, seg_pos={0 0}, upload=56433]: Done uploading piece. err: <nil> Jul 26 22:26:04.859459182 (58h50m36.049757755s) [noise=true , address=79.127.205.245:4252 , piece_id=7I4XQOUS4K4DLCN7ABMVYBTMMDLBMJ35N636J3PXIPQEXRPX2Q5A, seg_pos={0 0}, upload=56433]: Done uploading piece. err: <nil> Jul 26 22:26:04.870384477 (58h50m36.060683045s) [noise=true , address=104.36.82.55:4228 , piece_id=FDUWZQFR7KPANQ2Q7BBN35MAVMVPTIRAR43FKSIWWNFSEQEYJ2WQ, seg_pos={0 0}, upload=56433]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: storage node overloaded, request limit: 2000 Jul 26 22:26:04.876853131 (58h50m36.067151704s) [noise=true , address=79.127.205.246:4286 , piece_id=LLMIQNN2MC4CVQ3A2IKSCESOZQIKPVTNO7BU6GNGZDK6FYW3HW7A, seg_pos={0 0}, upload=56433]: Done uploading piece. err: <nil> Jul 26 22:26:04.896304375 (58h50m36.086602948s) [noise=true , address=156.146.43.225:4434 , piece_id=SF57F4DWPESIJDDRIIELIW6U2LWPRVDDP2ZAQ37PJRNKMJZP3SPQ, seg_pos={0 0}, upload=56433]: Done uploading piece. err: <nil> Jul 26 22:26:04.910010251 (58h50m36.10030884s) [noise=true , address=79.127.205.235:4680 , piece_id=DT7W7FPKQ2UM36C5BOD7EQCIIQD3T3RZIFTYL76CV7BY55MQDAFQ, seg_pos={0 0}, upload=56433]: Done uploading piece. err: <nil> Jul 26 22:26:05.570231715 (58h50m36.76053028s) [noise=true , address=64.49.13.138:4482 , piece_id=OIYY6RWLKYXBIU6Y3V6V7PDC37YCNAWQTB2OWZ4ZEQ4P7YMC6OQQ, seg_pos={0 0}, upload=56433]: Done uploading piece. err: <nil> Jul 26 22:26:09.837451748 (58h50m41.027750321s) [noise=true , address=77.90.31.18:4821 , piece_id=AYMNMUPFUHZFB6NBKFJIAZJQCJBEIKZB43WUIIAC7HED7XMDD4YQ, seg_pos={0 0}, upload=56433]: Done uploading piece. err: <nil> Jul 26 22:26:17.858065621 (58h50m49.048364194s) [noise=true , address=104.36.82.56:4236 , piece_id=UOIFGBYHYQBU3H255M2IBSG337FA2OIO63PLIR2IO5ROCT22245A, seg_pos={0 0}, upload=56433]: Done uploading piece. err: <nil> Jul 26 22:26:20.682613085 (58h50m51.872911678s) [noise=true , address=79.127.201.215:4350 , piece_id=SLAQNUXNJJTICL6PWWD6VBNMOCDNODPDCKQSLHESIJITWNNNWZSA, seg_pos={0 0}, upload=56433]: Done uploading piece. err: <nil> Jul 26 22:26:23.635267030 (58h50m54.825565598s) [noise=true , address=104.36.82.52:4087 , piece_id=QNQPJ76ECSGCZAIKVDZPMVNTRGBEZUQC7XA4OL7EQ2NWMIZFSTUQ, seg_pos={0 0}, upload=56433]: Done uploading piece. err: <nil> Jul 26 22:26:42.858044125 (58h51m14.048342696s) [noise=true , address=104.36.82.12:4169 , piece_id=XVZ7WJNU6PGADQPHQ36BJANJPIRF4FERZ7ZRWMWTSWK6PGZKYWJQ, seg_pos={0 0}, upload=56433]: Done uploading piece. err: <nil> Jul 26 22:27:02.945644569 (58h51m34.135943162s) [noise=true , address=104.36.82.7:4921 , piece_id=D7RSTDEGZQQ67IPAW4UIKDOWR2JEDLMYNGI62XYXY2NGNFRYAMOQ, seg_pos={0 0}, upload=56433]: Done uploading piece. err: ecclient: upload failed (node:12QKJQDHC2wE9GjN8T92Ah33d6dyA2z3tqM1jg8tUp3J7B8cHww, address:104.36.82.7:4921): protocol: expected piece hash; write tcp 139.59.144.22:60212->104.36.82.7:4921: use of closed network connection; write tcp 139.59.144.22:60212->104.36.82.7:4921: use of closed network connection; EOF Jul 26 22:27:02.945789983 (58h51m34.136088553s) [noise=true , address=104.36.82.4:4828 , piece_id=D4A7YOMNKUK2P7EFJ7TWBJMAB4XVNUKXFSAIE522IKTQ6QMB3P3Q, seg_pos={0 0}, upload=56433]: Done uploading piece. err: ecclient: upload failed (node:12MV49LDqJJn8S7nLwgVSpaYiGKa3BZJYbbge8oRuiXQfbNecaJ, address:104.36.82.4:4828): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:27:02.945887395 (58h51m34.136185975s) [noise=true , address=77.90.31.12:4615 , piece_id=FCFU2LUVQGZ73JKAEFO677L36LQHEXXDKHB2G5NOV5NMZ624FADA, seg_pos={0 0}, upload=56433]: Done uploading piece. err: ecclient: upload failed (node:1xaory9AHtW4ra15wmL2CagPKKJU2u9keFo1ntBMy8AzjpkmwR, address:77.90.31.12:4615): protocol: write tcp 139.59.144.22:38418->77.90.31.12:4615: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:02.945978599 (58h51m34.136277188s) [noise=true , address=104.36.82.21:4487 , piece_id=JCXM5YBQNM7POAGPCG4MN4VPIGLTEOGDNBO3RFCIED3Z46HGUL2Q, seg_pos={0 0}, upload=56433]: Done uploading piece. err: ecclient: upload failed (node:19xsJiapoS3bYRaeQoZ3w6U9eKWTkGpncNPifT9MAqnxCeL7yK, address:104.36.82.21:4487): protocol: write tcp 139.59.144.22:50202->104.36.82.21:4487: use of closed network connection; write tcp 139.59.144.22:50202->104.36.82.21:4487: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50202->104.36.82.21:4487: use of closed network connection Jul 26 22:27:02.946042869 (58h51m34.136341461s) [seg_pos={0 0}, upload=56433]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:27:03.051393829 (58h51m34.241692418s) [upload=56433]: Done uploading segments... Jul 26 22:27:03.051448631 (58h51m34.24174722s) [upload=56433]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.108454705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........XX.X..................X.X.........................] duration: 2.268097956s FAIL (2025-07-26 22:26:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.268147366s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.016666506s FAIL (2025-07-26 22:26:44 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 1.016697395s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........XX.X..................X.X........................] duration: 2.327581773s FAIL (2025-07-26 22:26:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.327614566s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X.......X..X..] duration: 13.394132336s FAIL (2025-07-26 22:26:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.394159207s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.190240947s FAIL (2025-07-26 22:26:24 UTC) orbiter warning: context deadline exceeded Duration: 3.190343499s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..................................X.............XX...XXXXX.X] duration: 1m0.008727547s FAIL (2025-07-26 22:26:02 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 437 (58h49m34.267255023s) [upload=56417]: Uploading segments... Jul 26 22:25:03.077269832 (58h49m34.267568403s) [upload=56417]: Got next segment. Inline: false Jul 26 22:25:03.077337050 (58h49m34.267635765s) [upload=56417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:03.077345533 (58h49m34.267644121s) [upload=56417]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:03.278398448 (58h49m34.468697055s) [seg_pos={0 0}, upload=56417]: Begin upload segment... Jul 26 22:25:03.278481989 (58h49m34.468780592s) [seg_pos={0 0}, upload=56417]: Done begin upload segment. Jul 26 22:25:03.278533497 (58h49m34.468832116s) [noise=true , address=104.36.82.52:4085 , piece_id=X47VSATIKUNVL45DF5G7TXWBHJQVDZLDBE5MITDN2JTS2LBDZFNA, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.279168913 (58h49m34.469467505s) [noise=true , address=45.91.116.12:4229 , piece_id=QD2BDJDMJPVVTAFHIODB44RYTGX4PXSNVIVX5QCBUKOZNX7GMCKA, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.279590873 (58h49m34.469889476s) [noise=true , address=104.36.82.55:4228 , piece_id=3SNM7NRDOPX7RNSFL5RGPP4FYCOJYSN6G33QYXOZ5EU576U7MFAQ, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.279979681 (58h49m34.470278275s) [noise=true , address=104.36.82.51:4084 , piece_id=K5QQXDOBZEG42VQGVNWCWR7HUYJHOJD5PFKQFZ375GXYUASZJEVQ, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.280479666 (58h49m34.470778259s) [noise=true , address=79.127.205.246:4275 , piece_id=X5BCWXVK2AENBQ67S4ZOGGW4HP3EAAJU2SUYKHCJBKHKQKYKWU7A, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.280857704 (58h49m34.471156298s) [noise=true , address=79.127.201.215:4359 , piece_id=BKZCFM3FDJAGHYOBZI7WHLMD336N6NAQQCPZA6N3HBV5J3T5PLQQ, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.281213825 (58h49m34.471512417s) [noise=true , address=104.36.82.56:4234 , piece_id=MJ5Y6LWBVSS2O77VFZ67KE6NPYDX7H2YQAEYBFHZPPQ74XEL3PSQ, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.281573738 (58h49m34.471872332s) [noise=true , address=79.127.205.245:4228 , piece_id=6E3TNQHI7V2LVEQG62F733ODM2WQA3LHQGXRL5PZHUK5OISIOVKA, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.281971944 (58h49m34.47227054s) [noise=true , address=156.146.43.225:4434 , piece_id=DLNLVAFQSBCQB4OIKX5I456T35DT5PVPHVFCXVNPOKPU2PRBLTGA, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.282339545 (58h49m34.472638139s) [noise=true , address=79.127.205.235:4690 , piece_id=DVSEEGF6CCKB645DNNKX34Q44VB54YA5JYLNPTZNPEZJRJD6HQEA, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.282715364 (58h49m34.473013958s) [noise=true , address=104.36.82.43:4868 , piece_id=QRI5IUCJWAZM4ZUA3AB7YE6WKMKBGC3GWBDZ3OP5RMYZLB76PBWA, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.283279641 (58h49m34.473578235s) [noise=true , address=79.127.219.47:4061 , piece_id=XQPEAJPN2YDGAJMAJ476XZ4R5XB2C5W54UTO5PIPO2PBGC4EM7BA, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.283655216 (58h49m34.473953818s) [noise=true , address=79.127.205.234:4601 , piece_id=S7HLK43SSKUJACGUMSBJD5RYVQ7PHVL642HDRXUB5HBYLGLHJMZQ, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.284029963 (58h49m34.474328556s) [noise=true , address=79.127.219.51:4210 , piece_id=KGNLSNBQAE7F7KOP4ZWQYMEFPZKEVZF4EURX5TV2DMG3LLB5RRXA, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.284403969 (58h49m34.47470256s) [noise=true , address=104.36.82.11:4130 , piece_id=ELGSFXADMVVURLHL5FNIQJSXKQTHEPZDPYPIZMZ5LUVECU2OISMA, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.284760537 (58h49m34.475059129s) [noise=true , address=64.49.13.138:4482 , piece_id=VRQDXJIJKMT73VKMTN2GFXC6I74YDP2RDWRGWI7CNH6PAUFCQFXQ, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.285097297 (58h49m34.475395893s) [noise=true , address=79.127.205.233:4633 , piece_id=VF77JFN6ZLHU4OJPS3C622HSTNLOPKEVWS4GQJ6VVCEPHITT27RQ, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.285447412 (58h49m34.475746006s) [noise=true , address=77.90.31.16:4748 , piece_id=MHTO6I4BBPRUKUU6S7BD5SFAJPEQHZCK5YDPQHEV4O6SP5LZUR4Q, seg_pos={0 0}, upload=56417]: Uploading piece... Jul 26 22:25:03.842646563 (58h49m35.032945152s) [noise=true , address=79.127.219.47:4061 , piece_id=XQPEAJPN2YDGAJMAJ476XZ4R5XB2C5W54UTO5PIPO2PBGC4EM7BA, seg_pos={0 0}, upload=56417]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:25:03.957698357 (58h49m35.14799693s) [upload=56417]: Next returned nil segment Jul 26 22:25:03.958303154 (58h49m35.148601742s) [upload=56417]: Waiting for error group managing segments... Jul 26 22:25:04.118496349 (58h49m35.308794923s) [noise=true , address=79.127.219.51:4210 , piece_id=KGNLSNBQAE7F7KOP4ZWQYMEFPZKEVZF4EURX5TV2DMG3LLB5RRXA, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:04.486857161 (58h49m35.677155748s) [noise=true , address=104.36.82.52:4085 , piece_id=X47VSATIKUNVL45DF5G7TXWBHJQVDZLDBE5MITDN2JTS2LBDZFNA, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:04.603543397 (58h49m35.79384197s) [noise=true , address=79.127.205.234:4601 , piece_id=S7HLK43SSKUJACGUMSBJD5RYVQ7PHVL642HDRXUB5HBYLGLHJMZQ, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:04.620749151 (58h49m35.811047723s) [noise=true , address=79.127.205.246:4275 , piece_id=X5BCWXVK2AENBQ67S4ZOGGW4HP3EAAJU2SUYKHCJBKHKQKYKWU7A, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:04.655159090 (58h49m35.845457663s) [noise=true , address=79.127.205.245:4228 , piece_id=6E3TNQHI7V2LVEQG62F733ODM2WQA3LHQGXRL5PZHUK5OISIOVKA, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:04.676357262 (58h49m35.866655849s) [noise=true , address=79.127.205.235:4690 , piece_id=DVSEEGF6CCKB645DNNKX34Q44VB54YA5JYLNPTZNPEZJRJD6HQEA, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:04.682903879 (58h49m35.873202452s) [noise=true , address=79.127.205.233:4633 , piece_id=VF77JFN6ZLHU4OJPS3C622HSTNLOPKEVWS4GQJ6VVCEPHITT27RQ, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:04.733223090 (58h49m35.923521669s) [noise=true , address=64.49.13.138:4482 , piece_id=VRQDXJIJKMT73VKMTN2GFXC6I74YDP2RDWRGWI7CNH6PAUFCQFXQ, seg_pos={0 0}, upload=56417]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:25:04.953275466 (58h49m36.143574046s) [noise=true , address=104.36.82.55:4228 , piece_id=3SNM7NRDOPX7RNSFL5RGPP4FYCOJYSN6G33QYXOZ5EU576U7MFAQ, seg_pos={0 0}, upload=56417]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: storage node overloaded, request limit: 2000 Jul 26 22:25:07.323648840 (58h49m38.513947412s) [noise=true , address=79.127.201.215:4359 , piece_id=BKZCFM3FDJAGHYOBZI7WHLMD336N6NAQQCPZA6N3HBV5J3T5PLQQ, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:07.713900478 (58h49m38.904199046s) [noise=true , address=45.91.116.12:4229 , piece_id=QD2BDJDMJPVVTAFHIODB44RYTGX4PXSNVIVX5QCBUKOZNX7GMCKA, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:10.979322525 (58h49m42.169621105s) [noise=true , address=156.146.43.225:4434 , piece_id=DLNLVAFQSBCQB4OIKX5I456T35DT5PVPHVFCXVNPOKPU2PRBLTGA, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:18.183631243 (58h49m49.373929833s) [noise=true , address=104.36.82.56:4234 , piece_id=MJ5Y6LWBVSS2O77VFZ67KE6NPYDX7H2YQAEYBFHZPPQ74XEL3PSQ, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:18.404493082 (58h49m49.594791721s) [noise=true , address=77.90.31.16:4748 , piece_id=MHTO6I4BBPRUKUU6S7BD5SFAJPEQHZCK5YDPQHEV4O6SP5LZUR4Q, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:43.960891516 (58h50m15.151190088s) [noise=true , address=104.36.82.11:4130 , piece_id=ELGSFXADMVVURLHL5FNIQJSXKQTHEPZDPYPIZMZ5LUVECU2OISMA, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:55.267676249 (58h50m26.457974821s) [noise=true , address=104.36.82.51:4084 , piece_id=K5QQXDOBZEG42VQGVNWCWR7HUYJHOJD5PFKQFZ375GXYUASZJEVQ, seg_pos={0 0}, upload=56417]: Done uploading piece. err: <nil> Jul 26 22:25:55.267707162 (58h50m26.458005742s) [seg_pos={0 0}, upload=56417]: Segment reached optimal threshold of 14 pieces. Jul 26 22:25:55.267884819 (58h50m26.458183399s) [noise=true , address=104.36.82.43:4868 , piece_id=QRI5IUCJWAZM4ZUA3AB7YE6WKMKBGC3GWBDZ3OP5RMYZLB76PBWA, seg_pos={0 0}, upload=56417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LJtYshPwu1uZJwVyjrjUMk3htyVnp9ntnZWSUb7X3bNkwU6t): protocol: write tcp 139.59.144.22:42408->104.36.82.43:4868: use of closed network connection; write tcp 139.59.144.22:42408->104.36.82.43:4868: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42408->104.36.82.43:4868: use of closed network connection Jul 26 22:25:55.267905763 (58h50m26.458204341s) [seg_pos={0 0}, upload=56417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 26 22:25:55.267976791 (58h50m26.45827537s) [upload=56417]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:25:55.267983188 (58h50m26.458281767s) [upload=56417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:25:55.430334230 (58h50m26.620632809s) [upload=56417]: Done uploading segments... Jul 26 22:25:55.430353491 (58h50m26.62065207s) [upload=56417]: Upload finished. err: <nil> Duration: 1m0.008756113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..............X.X..X..XXX....X.X.X...............X..........] duration: 2.351530433s FAIL (2025-07-26 22:26:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.351582957s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.725087729s FAIL (2025-07-26 22:25:56 UTC) orbiter warning: context deadline exceeded Duration: 2.725156921s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 947.256656ms FAIL (2025-07-26 22:25:55 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 947.302838ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)