Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.571885081s FAIL (2025-10-06 22:27:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.571915164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.742128677s FAIL (2025-10-06 22:26:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.742187593s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 3.040902398s FAIL (2025-10-06 22:26:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.040928161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.69516674s FAIL (2025-10-06 22:26:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.695197526s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.921797256s FAIL (2025-10-06 22:26:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.921817693s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.883622382s FAIL (2025-10-06 22:25:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.883667229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.684363262s FAIL (2025-10-06 22:25:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.684394151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.732953798s FAIL (2025-10-06 22:24:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.732980403s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.972973806s FAIL (2025-10-06 22:24:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.973002426s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 3.013841424s FAIL (2025-10-06 22:24:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.01410101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.656233847s FAIL (2025-10-06 22:24:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.656263928s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.824226287s FAIL (2025-10-06 22:23:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.824292395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.729467756s FAIL (2025-10-06 22:22:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.729499406s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 3.079564388s FAIL (2025-10-06 22:22:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.079591262s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.604395287s FAIL (2025-10-06 22:21:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.604461467s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.738054255s FAIL (2025-10-06 22:21:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.738082476s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.932018885s FAIL (2025-10-06 22:20:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.932056582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.586439177s FAIL (2025-10-06 22:20:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.586525445s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.724870945s FAIL (2025-10-06 22:20:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.724904854s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 3.110316404s FAIL (2025-10-06 22:19:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.110376883s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.813319698s FAIL (2025-10-06 22:19:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.813348611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.722746967s FAIL (2025-10-06 22:18:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.722839614s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.787538143s FAIL (2025-10-06 22:18:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.787578125s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.926892142s FAIL (2025-10-06 22:18:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.926929369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.59816174s FAIL (2025-10-06 22:17:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.598195674s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.848833032s FAIL (2025-10-06 22:17:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.848858221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.740514944s FAIL (2025-10-06 22:17:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.740548991s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.663782776s FAIL (2025-10-06 22:16:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.663814045s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.846001787s FAIL (2025-10-06 22:16:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.846071061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.680838199s FAIL (2025-10-06 22:15:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.68087966s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.571857752s FAIL (2025-10-06 22:15:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.5718915s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.863864671s FAIL (2025-10-06 22:15:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.86389634s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.901739418s FAIL (2025-10-06 22:14:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.901768666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.594685284s FAIL (2025-10-06 22:14:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.59474098s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.950808468s FAIL (2025-10-06 22:14:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.951611933s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.843341141s FAIL (2025-10-06 22:13:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.843367551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 3.042406827s FAIL (2025-10-06 22:13:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.042436929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.741163007s FAIL (2025-10-06 22:13:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.741192385s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.983448673s FAIL (2025-10-06 22:13:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.983478887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 3.032726291s FAIL (2025-10-06 22:12:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.032761451s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.895588405s FAIL (2025-10-06 22:12:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.895612285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.654793129s FAIL (2025-10-06 22:11:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.654823336s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.874088829s FAIL (2025-10-06 22:11:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.874119429s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 3.296503799s FAIL (2025-10-06 22:10:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.296527838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.61110193s FAIL (2025-10-06 22:10:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.61113342s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.621566715s FAIL (2025-10-06 22:10:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.621620007s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 3.311279788s FAIL (2025-10-06 22:10:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.311396442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.696093103s FAIL (2025-10-06 22:09:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.696126363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.7315399s FAIL (2025-10-06 22:09:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments... Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment... Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment. Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece... Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments... Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil> Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments... Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.731574147s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.875477574s FAIL (2025-10-06 22:09:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.875503328s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)