Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X.X.....X.X...XX....X..........X.......X....X.X.......] duration: 2.933468649s FAIL (2025-09-22 03:10:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.933494443s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.X.XX.........X..X..X.X....X.....XX......X................] duration: 2.341786205s FAIL (2025-09-22 03:10:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.341825823s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X......X.X.....X.X...XX....X..........X.......X....X.X......] duration: 2.627883997s FAIL (2025-09-22 03:10:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.627918993s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X.X.XX.........X..X..X.X....X.....XX......X...............] duration: 2.445587664s FAIL (2025-09-22 03:09:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.445639976s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X......X.X.....X.X...XX....X..........X.......X....X.X.....] duration: 3.043781887s FAIL (2025-09-22 03:09:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.044264918s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.X.X.XX.........X..X..X.X....X.....XX......X..............] duration: 2.301254088s FAIL (2025-09-22 03:09:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.301286777s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X......X.X.....X.X...XX....X..........X.......X....X.X....] duration: 2.89235932s FAIL (2025-09-22 03:08:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.892396296s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X.X.X.XX.........X..X..X.X....X.....XX......X.............] duration: 2.421786244s FAIL (2025-09-22 03:08:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.421821015s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X..X......X.X.....X.X...XX....X..........X.......X....X.X...] duration: 2.917505262s FAIL (2025-09-22 03:08:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.917532165s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX.X.X.X.XX.........X..X..X.X....X.....XX......X............] duration: 2.500864676s FAIL (2025-09-22 03:08:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.500899511s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X..X......X.X.....X.X...XX....X..........X.......X....X.X..] duration: 2.665809211s FAIL (2025-09-22 03:07:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.665872855s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXX.X.X.X.XX.........X..X..X.X....X.....XX......X...........] duration: 2.36879721s FAIL (2025-09-22 03:07:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.368845013s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X..X......X.X.....X.X...XX....X..........X.......X....X.X.] duration: 2.66832083s FAIL (2025-09-22 03:06:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.668363958s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXXX.X.X.X.XX.........X..X..X.X....X.....XX......X..........] duration: 2.365953176s FAIL (2025-09-22 03:06:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.366008959s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X..X......X.X.....X.X...XX....X..........X.......X....X.X] duration: 30.001617205s FAIL (2025-09-22 03:06:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001643268s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXX.X.X.X.XX.........X..X..X.X....X.....XX......X.........] duration: 2.498583453s FAIL (2025-09-22 03:05:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.498679307s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X..X......X.X.....X.X...XX....X..........X.......X....X.] duration: 2.759148128s FAIL (2025-09-22 03:04:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.75917894s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXX.X.X.X.XX.........X..X..X.X....X.....XX......X........] duration: 2.55300958s FAIL (2025-09-22 03:04:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.5530579s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X..X......X.X.....X.X...XX....X..........X.......X....X] duration: 30.001411515s FAIL (2025-09-22 03:04:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001435603s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXXXXXX.X.X.X.XX.........X..X..X.X....X.....XX......X.......] duration: 2.39098322s FAIL (2025-09-22 03:03:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.391011288s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XXXXXXX.X.X.X.XX.........X..X..X.X....X.....XX......X......] duration: 2.504871561s FAIL (2025-09-22 03:03:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.504906473s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X..X......X.X.....X.X...XX....X..........X.......X....] duration: 2.826019318s FAIL (2025-09-22 03:03:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.826044197s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XXXXXXX.X.X.X.XX.........X..X..X.X....X.....XX......X.....] duration: 2.561477643s FAIL (2025-09-22 03:02:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.561506338s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......X..X......X.X.....X.X...XX....X..........X.......X...] duration: 2.778646927s FAIL (2025-09-22 03:02:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.77869972s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...XXXXXXX.X.X.X.XX.........X..X..X.X....X.....XX......X....] duration: 2.614533932s FAIL (2025-09-22 03:01:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.614573007s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.......X..X......X.X.....X.X...XX....X..........X.......X..] duration: 2.657159095s FAIL (2025-09-22 03:01:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.657186741s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X...XXXXXXX.X.X.X.XX.........X..X..X.X....X.....XX......X...] duration: 2.533458285s FAIL (2025-09-22 03:01:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.533495291s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X...XXXXXXX.X.X.X.XX.........X..X..X.X....X.....XX......X..] duration: 2.365704937s FAIL (2025-09-22 03:00:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.365749017s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.......X..X......X.X.....X.X...XX....X..........X.......X.] duration: 2.718370514s FAIL (2025-09-22 03:00:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.71840351s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X.......X..X......X.X.....X.X...XX....X..........X.......X] duration: 30.001445955s FAIL (2025-09-22 03:00:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001476478s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X...XXXXXXX.X.X.X.XX.........X..X..X.X....X.....XX......X.] duration: 2.423886252s FAIL (2025-09-22 02:59:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.423928397s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.X.......X..X......X.X.....X.X...XX....X..........X.......] duration: 2.862525005s FAIL (2025-09-22 02:59:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.862804896s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X...XXXXXXX.X.X.X.XX.........X..X..X.X....X.....XX......X] duration: 30.135209463s FAIL (2025-09-22 02:59:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.135227523s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X.X.......X..X......X.X.....X.X...XX....X..........X......] duration: 2.794510268s FAIL (2025-09-22 02:58:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.794537848s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.742539357s FAIL (2025-09-22 02:58:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.7425772s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.X...XXXXXXX.X.X.X.XX.........X..X..X.X....X.....XX......] duration: 2.586172995s FAIL (2025-09-22 02:58:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:53:16.803730112 (564h45m37.296162545s) [upload=496844]: Starting upload Sep 22 02:53:16.803799753 (564h45m37.296232188s) [upload=496844]: Done starting upload Sep 22 02:53:16.803915386 (564h45m37.296347821s) [upload=496844]: Uploading segments... Sep 22 02:53:16.804081266 (564h45m37.296513698s) [upload=496844]: Got next segment. Inline: false Sep 22 02:53:16.804136100 (564h45m37.296568535s) [upload=496844]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:53:16.804142089 (564h45m37.296574532s) [upload=496844]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:53:16.924693065 (564h45m37.417125514s) [seg_pos={0 0}, upload=496844]: Begin upload segment... Sep 22 02:53:16.924761853 (564h45m37.417194292s) [seg_pos={0 0}, upload=496844]: Done begin upload segment. Sep 22 02:53:16.924794582 (564h45m37.417227034s) [noise=true , address=45.91.116.9:4262 , piece_id=AN2YEHJFWO3GV36HVYEUEBYWX7APL4GI2JGZVLTVMQRZSR74AENA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.925328296 (564h45m37.417760741s) [noise=true , address=77.90.31.17:4770 , piece_id=OSZDK2HOGWIPYQHHY3AYZHTEPLJLKFJBGHVY5IMPEEQLL3BFN7PA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.925670244 (564h45m37.418102687s) [noise=true , address=45.91.116.14:4190 , piece_id=LRN66OHJO5ST7F57Q66SGTW5T7Q2N4IK52TZXYWA232BYL4DNSCQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926006094 (564h45m37.418438536s) [noise=true , address=45.91.116.8:4045 , piece_id=VQRXTUUXBGHHINC26X75NGG3OAGRSJOJ3Y65EAXB7ENUQWFE6WRQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926206123 (564h45m37.418638568s) [noise=true , address=77.90.31.17:4771 , piece_id=MVI3VRDUA5K5O4N5UP2DP5OLCCJC44P4RAR4FLXJ5ZS77D27KXGA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926834042 (564h45m37.419266491s) [noise=true , address=77.90.31.11:4555 , piece_id=BCTTNFWLDS3RA5XZRKJJIMRHGMXRFJMA2FUCB5EXSIVLRRZHESDA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927294956 (564h45m37.419727397s) [noise=true , address=45.91.116.6:4973 , piece_id=WUIM6UBQH7GEXD2BAVDDNYWF4GQFWWCL5MRXCKZ5UJS5W5634EBA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927573044 (564h45m37.420005486s) [noise=true , address=45.91.116.4:4937 , piece_id=HXJZJYSK3A2IAT3PTE5SA5UGEPCXATEJOKITCQ2D7CWTVPDGPEBA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927864772 (564h45m37.420297214s) [noise=true , address=45.91.116.5:4299 , piece_id=3IKQPV4BWIA2ON2GBOATUDK7LPZ3SVGIIV46QCSI5IGSO42SROPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928013610 (564h45m37.420446133s) [noise=true , address=45.91.116.10:4081 , piece_id=WDVMAL2L4I5MOUSECP6S4PXL3TK53SBRRLYAKWR2RXO7PWQLXJSA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928149679 (564h45m37.420582123s) [noise=true , address=77.90.31.19:4841 , piece_id=Z52MG5CYIAUXMV32PHBPLYKGNCSJJSEGAK4DHHN3LJMYFHOJIL5A, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928546383 (564h45m37.420978825s) [noise=true , address=145.79.227.11:4732 , piece_id=KD4CBSFZ7RLL3FBKZ4NQKLS7PXFNQKXXGAOKKG5MERP423UJHOPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928891033 (564h45m37.421323475s) [noise=true , address=145.79.227.4:4186 , piece_id=2LFV4O5E5ZHYN24SEDLHHO334CDAW4AW7PAN322QXT3H2SRFSLPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928931827 (564h45m37.421364267s) [noise=true , address=145.79.227.16:4909 , piece_id=2F6FGN2RYDM5IUBZ5AXMQE7RYK3NKBKLCMGJOQ474KGWBMMHK2RQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.929444932 (564h45m37.421877365s) [noise=true , address=45.91.116.11:4118 , piece_id=5K5IYEDQU3PSFWG2IWNKWFOHWMG6BAQCPLIJLDHQQWEJYRNZ2GGA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.929986031 (564h45m37.422418464s) [noise=true , address=145.79.227.18:4005 , piece_id=KXBNHWEZASVN6O4N76LRA5Z7EYGKSGJZWXX26FGJC5OZD5CVO6AA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:17.532291405 (564h45m38.024723835s) [upload=496844]: Next returned nil segment Sep 22 02:53:17.532308544 (564h45m38.024740987s) [upload=496844]: Waiting for error group managing segments... Sep 22 02:53:17.673641008 (564h45m38.166073489s) [noise=true , address=45.91.116.9:4262 , piece_id=AN2YEHJFWO3GV36HVYEUEBYWX7APL4GI2JGZVLTVMQRZSR74AENA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.677178652 (564h45m38.169611082s) [noise=true , address=45.91.116.10:4081 , piece_id=WDVMAL2L4I5MOUSECP6S4PXL3TK53SBRRLYAKWR2RXO7PWQLXJSA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.678967137 (564h45m38.171399573s) [noise=true , address=45.91.116.4:4937 , piece_id=HXJZJYSK3A2IAT3PTE5SA5UGEPCXATEJOKITCQ2D7CWTVPDGPEBA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.681878108 (564h45m38.174310536s) [noise=true , address=45.91.116.11:4118 , piece_id=5K5IYEDQU3PSFWG2IWNKWFOHWMG6BAQCPLIJLDHQQWEJYRNZ2GGA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.686890524 (564h45m38.179322957s) [noise=true , address=77.90.31.11:4555 , piece_id=BCTTNFWLDS3RA5XZRKJJIMRHGMXRFJMA2FUCB5EXSIVLRRZHESDA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.686940587 (564h45m38.179373029s) [noise=true , address=77.90.31.19:4841 , piece_id=Z52MG5CYIAUXMV32PHBPLYKGNCSJJSEGAK4DHHN3LJMYFHOJIL5A, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687424121 (564h45m38.179856556s) [noise=true , address=77.90.31.17:4770 , piece_id=OSZDK2HOGWIPYQHHY3AYZHTEPLJLKFJBGHVY5IMPEEQLL3BFN7PA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687535995 (564h45m38.179968424s) [noise=true , address=45.91.116.14:4190 , piece_id=LRN66OHJO5ST7F57Q66SGTW5T7Q2N4IK52TZXYWA232BYL4DNSCQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687981077 (564h45m38.180413511s) [noise=true , address=77.90.31.17:4771 , piece_id=MVI3VRDUA5K5O4N5UP2DP5OLCCJC44P4RAR4FLXJ5ZS77D27KXGA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.688037279 (564h45m38.180469715s) [noise=true , address=45.91.116.8:4045 , piece_id=VQRXTUUXBGHHINC26X75NGG3OAGRSJOJ3Y65EAXB7ENUQWFE6WRQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.690583072 (564h45m38.183015501s) [noise=true , address=45.91.116.6:4973 , piece_id=WUIM6UBQH7GEXD2BAVDDNYWF4GQFWWCL5MRXCKZ5UJS5W5634EBA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.697337949 (564h45m38.189770384s) [noise=true , address=45.91.116.5:4299 , piece_id=3IKQPV4BWIA2ON2GBOATUDK7LPZ3SVGIIV46QCSI5IGSO42SROPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.901510874 (564h45m38.393943323s) [noise=true , address=145.79.227.11:4732 , piece_id=KD4CBSFZ7RLL3FBKZ4NQKLS7PXFNQKXXGAOKKG5MERP423UJHOPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499213541 (564h46m6.991645996s) [noise=true , address=145.79.227.4:4186 , piece_id=2LFV4O5E5ZHYN24SEDLHHO334CDAW4AW7PAN322QXT3H2SRFSLPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499294035 (564h46m6.99172648s) [noise=true , address=145.79.227.16:4909 , piece_id=2F6FGN2RYDM5IUBZ5AXMQE7RYK3NKBKLCMGJOQ474KGWBMMHK2RQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499316904 (564h46m6.991749347s) [noise=true , address=145.79.227.18:4005 , piece_id=KXBNHWEZASVN6O4N76LRA5Z7EYGKSGJZWXX26FGJC5OZD5CVO6AA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499374361 (564h46m6.99180681s) [seg_pos={0 0}, upload=496844]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:53:46.611368399 (564h46m7.103800835s) [upload=496844]: Done uploading segments... Sep 22 02:53:46.611396228 (564h46m7.103828662s) [upload=496844]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.586226264s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.730367179s FAIL (2025-09-22 02:58:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.730470522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X.X...XXXXXXX.X.X.X.XX.........X..X..X.X....X.....XX.....] duration: 2.58022122s FAIL (2025-09-22 02:57:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:53:16.803730112 (564h45m37.296162545s) [upload=496844]: Starting upload Sep 22 02:53:16.803799753 (564h45m37.296232188s) [upload=496844]: Done starting upload Sep 22 02:53:16.803915386 (564h45m37.296347821s) [upload=496844]: Uploading segments... Sep 22 02:53:16.804081266 (564h45m37.296513698s) [upload=496844]: Got next segment. Inline: false Sep 22 02:53:16.804136100 (564h45m37.296568535s) [upload=496844]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:53:16.804142089 (564h45m37.296574532s) [upload=496844]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:53:16.924693065 (564h45m37.417125514s) [seg_pos={0 0}, upload=496844]: Begin upload segment... Sep 22 02:53:16.924761853 (564h45m37.417194292s) [seg_pos={0 0}, upload=496844]: Done begin upload segment. Sep 22 02:53:16.924794582 (564h45m37.417227034s) [noise=true , address=45.91.116.9:4262 , piece_id=AN2YEHJFWO3GV36HVYEUEBYWX7APL4GI2JGZVLTVMQRZSR74AENA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.925328296 (564h45m37.417760741s) [noise=true , address=77.90.31.17:4770 , piece_id=OSZDK2HOGWIPYQHHY3AYZHTEPLJLKFJBGHVY5IMPEEQLL3BFN7PA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.925670244 (564h45m37.418102687s) [noise=true , address=45.91.116.14:4190 , piece_id=LRN66OHJO5ST7F57Q66SGTW5T7Q2N4IK52TZXYWA232BYL4DNSCQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926006094 (564h45m37.418438536s) [noise=true , address=45.91.116.8:4045 , piece_id=VQRXTUUXBGHHINC26X75NGG3OAGRSJOJ3Y65EAXB7ENUQWFE6WRQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926206123 (564h45m37.418638568s) [noise=true , address=77.90.31.17:4771 , piece_id=MVI3VRDUA5K5O4N5UP2DP5OLCCJC44P4RAR4FLXJ5ZS77D27KXGA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926834042 (564h45m37.419266491s) [noise=true , address=77.90.31.11:4555 , piece_id=BCTTNFWLDS3RA5XZRKJJIMRHGMXRFJMA2FUCB5EXSIVLRRZHESDA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927294956 (564h45m37.419727397s) [noise=true , address=45.91.116.6:4973 , piece_id=WUIM6UBQH7GEXD2BAVDDNYWF4GQFWWCL5MRXCKZ5UJS5W5634EBA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927573044 (564h45m37.420005486s) [noise=true , address=45.91.116.4:4937 , piece_id=HXJZJYSK3A2IAT3PTE5SA5UGEPCXATEJOKITCQ2D7CWTVPDGPEBA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927864772 (564h45m37.420297214s) [noise=true , address=45.91.116.5:4299 , piece_id=3IKQPV4BWIA2ON2GBOATUDK7LPZ3SVGIIV46QCSI5IGSO42SROPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928013610 (564h45m37.420446133s) [noise=true , address=45.91.116.10:4081 , piece_id=WDVMAL2L4I5MOUSECP6S4PXL3TK53SBRRLYAKWR2RXO7PWQLXJSA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928149679 (564h45m37.420582123s) [noise=true , address=77.90.31.19:4841 , piece_id=Z52MG5CYIAUXMV32PHBPLYKGNCSJJSEGAK4DHHN3LJMYFHOJIL5A, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928546383 (564h45m37.420978825s) [noise=true , address=145.79.227.11:4732 , piece_id=KD4CBSFZ7RLL3FBKZ4NQKLS7PXFNQKXXGAOKKG5MERP423UJHOPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928891033 (564h45m37.421323475s) [noise=true , address=145.79.227.4:4186 , piece_id=2LFV4O5E5ZHYN24SEDLHHO334CDAW4AW7PAN322QXT3H2SRFSLPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928931827 (564h45m37.421364267s) [noise=true , address=145.79.227.16:4909 , piece_id=2F6FGN2RYDM5IUBZ5AXMQE7RYK3NKBKLCMGJOQ474KGWBMMHK2RQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.929444932 (564h45m37.421877365s) [noise=true , address=45.91.116.11:4118 , piece_id=5K5IYEDQU3PSFWG2IWNKWFOHWMG6BAQCPLIJLDHQQWEJYRNZ2GGA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.929986031 (564h45m37.422418464s) [noise=true , address=145.79.227.18:4005 , piece_id=KXBNHWEZASVN6O4N76LRA5Z7EYGKSGJZWXX26FGJC5OZD5CVO6AA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:17.532291405 (564h45m38.024723835s) [upload=496844]: Next returned nil segment Sep 22 02:53:17.532308544 (564h45m38.024740987s) [upload=496844]: Waiting for error group managing segments... Sep 22 02:53:17.673641008 (564h45m38.166073489s) [noise=true , address=45.91.116.9:4262 , piece_id=AN2YEHJFWO3GV36HVYEUEBYWX7APL4GI2JGZVLTVMQRZSR74AENA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.677178652 (564h45m38.169611082s) [noise=true , address=45.91.116.10:4081 , piece_id=WDVMAL2L4I5MOUSECP6S4PXL3TK53SBRRLYAKWR2RXO7PWQLXJSA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.678967137 (564h45m38.171399573s) [noise=true , address=45.91.116.4:4937 , piece_id=HXJZJYSK3A2IAT3PTE5SA5UGEPCXATEJOKITCQ2D7CWTVPDGPEBA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.681878108 (564h45m38.174310536s) [noise=true , address=45.91.116.11:4118 , piece_id=5K5IYEDQU3PSFWG2IWNKWFOHWMG6BAQCPLIJLDHQQWEJYRNZ2GGA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.686890524 (564h45m38.179322957s) [noise=true , address=77.90.31.11:4555 , piece_id=BCTTNFWLDS3RA5XZRKJJIMRHGMXRFJMA2FUCB5EXSIVLRRZHESDA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.686940587 (564h45m38.179373029s) [noise=true , address=77.90.31.19:4841 , piece_id=Z52MG5CYIAUXMV32PHBPLYKGNCSJJSEGAK4DHHN3LJMYFHOJIL5A, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687424121 (564h45m38.179856556s) [noise=true , address=77.90.31.17:4770 , piece_id=OSZDK2HOGWIPYQHHY3AYZHTEPLJLKFJBGHVY5IMPEEQLL3BFN7PA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687535995 (564h45m38.179968424s) [noise=true , address=45.91.116.14:4190 , piece_id=LRN66OHJO5ST7F57Q66SGTW5T7Q2N4IK52TZXYWA232BYL4DNSCQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687981077 (564h45m38.180413511s) [noise=true , address=77.90.31.17:4771 , piece_id=MVI3VRDUA5K5O4N5UP2DP5OLCCJC44P4RAR4FLXJ5ZS77D27KXGA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.688037279 (564h45m38.180469715s) [noise=true , address=45.91.116.8:4045 , piece_id=VQRXTUUXBGHHINC26X75NGG3OAGRSJOJ3Y65EAXB7ENUQWFE6WRQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.690583072 (564h45m38.183015501s) [noise=true , address=45.91.116.6:4973 , piece_id=WUIM6UBQH7GEXD2BAVDDNYWF4GQFWWCL5MRXCKZ5UJS5W5634EBA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.697337949 (564h45m38.189770384s) [noise=true , address=45.91.116.5:4299 , piece_id=3IKQPV4BWIA2ON2GBOATUDK7LPZ3SVGIIV46QCSI5IGSO42SROPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.901510874 (564h45m38.393943323s) [noise=true , address=145.79.227.11:4732 , piece_id=KD4CBSFZ7RLL3FBKZ4NQKLS7PXFNQKXXGAOKKG5MERP423UJHOPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499213541 (564h46m6.991645996s) [noise=true , address=145.79.227.4:4186 , piece_id=2LFV4O5E5ZHYN24SEDLHHO334CDAW4AW7PAN322QXT3H2SRFSLPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499294035 (564h46m6.99172648s) [noise=true , address=145.79.227.16:4909 , piece_id=2F6FGN2RYDM5IUBZ5AXMQE7RYK3NKBKLCMGJOQ474KGWBMMHK2RQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499316904 (564h46m6.991749347s) [noise=true , address=145.79.227.18:4005 , piece_id=KXBNHWEZASVN6O4N76LRA5Z7EYGKSGJZWXX26FGJC5OZD5CVO6AA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499374361 (564h46m6.99180681s) [seg_pos={0 0}, upload=496844]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:53:46.611368399 (564h46m7.103800835s) [upload=496844]: Done uploading segments... Sep 22 02:53:46.611396228 (564h46m7.103828662s) [upload=496844]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.580319676s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.738097555s FAIL (2025-09-22 02:57:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.738125714s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.X.......X..X......X.X.....X.X...XX....X..........X.....] duration: 2.889622195s FAIL (2025-09-22 02:57:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.88964382s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.625351719s FAIL (2025-09-22 02:56:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.625380116s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.X.......X..X......X.X.....X.X...XX....X..........X....] duration: 2.912630707s FAIL (2025-09-22 02:56:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.912664104s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX..X.X...XXXXXXX.X.X.X.XX.........X..X..X.X....X.....XX....] duration: 2.625282047s FAIL (2025-09-22 02:56:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:53:16.803730112 (564h45m37.296162545s) [upload=496844]: Starting upload Sep 22 02:53:16.803799753 (564h45m37.296232188s) [upload=496844]: Done starting upload Sep 22 02:53:16.803915386 (564h45m37.296347821s) [upload=496844]: Uploading segments... Sep 22 02:53:16.804081266 (564h45m37.296513698s) [upload=496844]: Got next segment. Inline: false Sep 22 02:53:16.804136100 (564h45m37.296568535s) [upload=496844]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:53:16.804142089 (564h45m37.296574532s) [upload=496844]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:53:16.924693065 (564h45m37.417125514s) [seg_pos={0 0}, upload=496844]: Begin upload segment... Sep 22 02:53:16.924761853 (564h45m37.417194292s) [seg_pos={0 0}, upload=496844]: Done begin upload segment. Sep 22 02:53:16.924794582 (564h45m37.417227034s) [noise=true , address=45.91.116.9:4262 , piece_id=AN2YEHJFWO3GV36HVYEUEBYWX7APL4GI2JGZVLTVMQRZSR74AENA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.925328296 (564h45m37.417760741s) [noise=true , address=77.90.31.17:4770 , piece_id=OSZDK2HOGWIPYQHHY3AYZHTEPLJLKFJBGHVY5IMPEEQLL3BFN7PA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.925670244 (564h45m37.418102687s) [noise=true , address=45.91.116.14:4190 , piece_id=LRN66OHJO5ST7F57Q66SGTW5T7Q2N4IK52TZXYWA232BYL4DNSCQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926006094 (564h45m37.418438536s) [noise=true , address=45.91.116.8:4045 , piece_id=VQRXTUUXBGHHINC26X75NGG3OAGRSJOJ3Y65EAXB7ENUQWFE6WRQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926206123 (564h45m37.418638568s) [noise=true , address=77.90.31.17:4771 , piece_id=MVI3VRDUA5K5O4N5UP2DP5OLCCJC44P4RAR4FLXJ5ZS77D27KXGA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926834042 (564h45m37.419266491s) [noise=true , address=77.90.31.11:4555 , piece_id=BCTTNFWLDS3RA5XZRKJJIMRHGMXRFJMA2FUCB5EXSIVLRRZHESDA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927294956 (564h45m37.419727397s) [noise=true , address=45.91.116.6:4973 , piece_id=WUIM6UBQH7GEXD2BAVDDNYWF4GQFWWCL5MRXCKZ5UJS5W5634EBA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927573044 (564h45m37.420005486s) [noise=true , address=45.91.116.4:4937 , piece_id=HXJZJYSK3A2IAT3PTE5SA5UGEPCXATEJOKITCQ2D7CWTVPDGPEBA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927864772 (564h45m37.420297214s) [noise=true , address=45.91.116.5:4299 , piece_id=3IKQPV4BWIA2ON2GBOATUDK7LPZ3SVGIIV46QCSI5IGSO42SROPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928013610 (564h45m37.420446133s) [noise=true , address=45.91.116.10:4081 , piece_id=WDVMAL2L4I5MOUSECP6S4PXL3TK53SBRRLYAKWR2RXO7PWQLXJSA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928149679 (564h45m37.420582123s) [noise=true , address=77.90.31.19:4841 , piece_id=Z52MG5CYIAUXMV32PHBPLYKGNCSJJSEGAK4DHHN3LJMYFHOJIL5A, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928546383 (564h45m37.420978825s) [noise=true , address=145.79.227.11:4732 , piece_id=KD4CBSFZ7RLL3FBKZ4NQKLS7PXFNQKXXGAOKKG5MERP423UJHOPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928891033 (564h45m37.421323475s) [noise=true , address=145.79.227.4:4186 , piece_id=2LFV4O5E5ZHYN24SEDLHHO334CDAW4AW7PAN322QXT3H2SRFSLPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928931827 (564h45m37.421364267s) [noise=true , address=145.79.227.16:4909 , piece_id=2F6FGN2RYDM5IUBZ5AXMQE7RYK3NKBKLCMGJOQ474KGWBMMHK2RQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.929444932 (564h45m37.421877365s) [noise=true , address=45.91.116.11:4118 , piece_id=5K5IYEDQU3PSFWG2IWNKWFOHWMG6BAQCPLIJLDHQQWEJYRNZ2GGA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.929986031 (564h45m37.422418464s) [noise=true , address=145.79.227.18:4005 , piece_id=KXBNHWEZASVN6O4N76LRA5Z7EYGKSGJZWXX26FGJC5OZD5CVO6AA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:17.532291405 (564h45m38.024723835s) [upload=496844]: Next returned nil segment Sep 22 02:53:17.532308544 (564h45m38.024740987s) [upload=496844]: Waiting for error group managing segments... Sep 22 02:53:17.673641008 (564h45m38.166073489s) [noise=true , address=45.91.116.9:4262 , piece_id=AN2YEHJFWO3GV36HVYEUEBYWX7APL4GI2JGZVLTVMQRZSR74AENA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.677178652 (564h45m38.169611082s) [noise=true , address=45.91.116.10:4081 , piece_id=WDVMAL2L4I5MOUSECP6S4PXL3TK53SBRRLYAKWR2RXO7PWQLXJSA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.678967137 (564h45m38.171399573s) [noise=true , address=45.91.116.4:4937 , piece_id=HXJZJYSK3A2IAT3PTE5SA5UGEPCXATEJOKITCQ2D7CWTVPDGPEBA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.681878108 (564h45m38.174310536s) [noise=true , address=45.91.116.11:4118 , piece_id=5K5IYEDQU3PSFWG2IWNKWFOHWMG6BAQCPLIJLDHQQWEJYRNZ2GGA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.686890524 (564h45m38.179322957s) [noise=true , address=77.90.31.11:4555 , piece_id=BCTTNFWLDS3RA5XZRKJJIMRHGMXRFJMA2FUCB5EXSIVLRRZHESDA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.686940587 (564h45m38.179373029s) [noise=true , address=77.90.31.19:4841 , piece_id=Z52MG5CYIAUXMV32PHBPLYKGNCSJJSEGAK4DHHN3LJMYFHOJIL5A, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687424121 (564h45m38.179856556s) [noise=true , address=77.90.31.17:4770 , piece_id=OSZDK2HOGWIPYQHHY3AYZHTEPLJLKFJBGHVY5IMPEEQLL3BFN7PA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687535995 (564h45m38.179968424s) [noise=true , address=45.91.116.14:4190 , piece_id=LRN66OHJO5ST7F57Q66SGTW5T7Q2N4IK52TZXYWA232BYL4DNSCQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687981077 (564h45m38.180413511s) [noise=true , address=77.90.31.17:4771 , piece_id=MVI3VRDUA5K5O4N5UP2DP5OLCCJC44P4RAR4FLXJ5ZS77D27KXGA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.688037279 (564h45m38.180469715s) [noise=true , address=45.91.116.8:4045 , piece_id=VQRXTUUXBGHHINC26X75NGG3OAGRSJOJ3Y65EAXB7ENUQWFE6WRQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.690583072 (564h45m38.183015501s) [noise=true , address=45.91.116.6:4973 , piece_id=WUIM6UBQH7GEXD2BAVDDNYWF4GQFWWCL5MRXCKZ5UJS5W5634EBA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.697337949 (564h45m38.189770384s) [noise=true , address=45.91.116.5:4299 , piece_id=3IKQPV4BWIA2ON2GBOATUDK7LPZ3SVGIIV46QCSI5IGSO42SROPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.901510874 (564h45m38.393943323s) [noise=true , address=145.79.227.11:4732 , piece_id=KD4CBSFZ7RLL3FBKZ4NQKLS7PXFNQKXXGAOKKG5MERP423UJHOPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499213541 (564h46m6.991645996s) [noise=true , address=145.79.227.4:4186 , piece_id=2LFV4O5E5ZHYN24SEDLHHO334CDAW4AW7PAN322QXT3H2SRFSLPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499294035 (564h46m6.99172648s) [noise=true , address=145.79.227.16:4909 , piece_id=2F6FGN2RYDM5IUBZ5AXMQE7RYK3NKBKLCMGJOQ474KGWBMMHK2RQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499316904 (564h46m6.991749347s) [noise=true , address=145.79.227.18:4005 , piece_id=KXBNHWEZASVN6O4N76LRA5Z7EYGKSGJZWXX26FGJC5OZD5CVO6AA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499374361 (564h46m6.99180681s) [seg_pos={0 0}, upload=496844]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:53:46.611368399 (564h46m7.103800835s) [upload=496844]: Done uploading segments... Sep 22 02:53:46.611396228 (564h46m7.103828662s) [upload=496844]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.625317166s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.X.......X..X......X.X.....X.X...XX....X..........X...] duration: 2.639366636s FAIL (2025-09-22 02:55:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.639422179s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.579810534s FAIL (2025-09-22 02:55:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.579949036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX..X.X...XXXXXXX.X.X.X.XX.........X..X..X.X....X.....XX...] duration: 2.415989673s FAIL (2025-09-22 02:55:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:53:16.803730112 (564h45m37.296162545s) [upload=496844]: Starting upload Sep 22 02:53:16.803799753 (564h45m37.296232188s) [upload=496844]: Done starting upload Sep 22 02:53:16.803915386 (564h45m37.296347821s) [upload=496844]: Uploading segments... Sep 22 02:53:16.804081266 (564h45m37.296513698s) [upload=496844]: Got next segment. Inline: false Sep 22 02:53:16.804136100 (564h45m37.296568535s) [upload=496844]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:53:16.804142089 (564h45m37.296574532s) [upload=496844]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:53:16.924693065 (564h45m37.417125514s) [seg_pos={0 0}, upload=496844]: Begin upload segment... Sep 22 02:53:16.924761853 (564h45m37.417194292s) [seg_pos={0 0}, upload=496844]: Done begin upload segment. Sep 22 02:53:16.924794582 (564h45m37.417227034s) [noise=true , address=45.91.116.9:4262 , piece_id=AN2YEHJFWO3GV36HVYEUEBYWX7APL4GI2JGZVLTVMQRZSR74AENA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.925328296 (564h45m37.417760741s) [noise=true , address=77.90.31.17:4770 , piece_id=OSZDK2HOGWIPYQHHY3AYZHTEPLJLKFJBGHVY5IMPEEQLL3BFN7PA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.925670244 (564h45m37.418102687s) [noise=true , address=45.91.116.14:4190 , piece_id=LRN66OHJO5ST7F57Q66SGTW5T7Q2N4IK52TZXYWA232BYL4DNSCQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926006094 (564h45m37.418438536s) [noise=true , address=45.91.116.8:4045 , piece_id=VQRXTUUXBGHHINC26X75NGG3OAGRSJOJ3Y65EAXB7ENUQWFE6WRQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926206123 (564h45m37.418638568s) [noise=true , address=77.90.31.17:4771 , piece_id=MVI3VRDUA5K5O4N5UP2DP5OLCCJC44P4RAR4FLXJ5ZS77D27KXGA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926834042 (564h45m37.419266491s) [noise=true , address=77.90.31.11:4555 , piece_id=BCTTNFWLDS3RA5XZRKJJIMRHGMXRFJMA2FUCB5EXSIVLRRZHESDA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927294956 (564h45m37.419727397s) [noise=true , address=45.91.116.6:4973 , piece_id=WUIM6UBQH7GEXD2BAVDDNYWF4GQFWWCL5MRXCKZ5UJS5W5634EBA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927573044 (564h45m37.420005486s) [noise=true , address=45.91.116.4:4937 , piece_id=HXJZJYSK3A2IAT3PTE5SA5UGEPCXATEJOKITCQ2D7CWTVPDGPEBA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927864772 (564h45m37.420297214s) [noise=true , address=45.91.116.5:4299 , piece_id=3IKQPV4BWIA2ON2GBOATUDK7LPZ3SVGIIV46QCSI5IGSO42SROPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928013610 (564h45m37.420446133s) [noise=true , address=45.91.116.10:4081 , piece_id=WDVMAL2L4I5MOUSECP6S4PXL3TK53SBRRLYAKWR2RXO7PWQLXJSA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928149679 (564h45m37.420582123s) [noise=true , address=77.90.31.19:4841 , piece_id=Z52MG5CYIAUXMV32PHBPLYKGNCSJJSEGAK4DHHN3LJMYFHOJIL5A, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928546383 (564h45m37.420978825s) [noise=true , address=145.79.227.11:4732 , piece_id=KD4CBSFZ7RLL3FBKZ4NQKLS7PXFNQKXXGAOKKG5MERP423UJHOPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928891033 (564h45m37.421323475s) [noise=true , address=145.79.227.4:4186 , piece_id=2LFV4O5E5ZHYN24SEDLHHO334CDAW4AW7PAN322QXT3H2SRFSLPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928931827 (564h45m37.421364267s) [noise=true , address=145.79.227.16:4909 , piece_id=2F6FGN2RYDM5IUBZ5AXMQE7RYK3NKBKLCMGJOQ474KGWBMMHK2RQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.929444932 (564h45m37.421877365s) [noise=true , address=45.91.116.11:4118 , piece_id=5K5IYEDQU3PSFWG2IWNKWFOHWMG6BAQCPLIJLDHQQWEJYRNZ2GGA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.929986031 (564h45m37.422418464s) [noise=true , address=145.79.227.18:4005 , piece_id=KXBNHWEZASVN6O4N76LRA5Z7EYGKSGJZWXX26FGJC5OZD5CVO6AA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:17.532291405 (564h45m38.024723835s) [upload=496844]: Next returned nil segment Sep 22 02:53:17.532308544 (564h45m38.024740987s) [upload=496844]: Waiting for error group managing segments... Sep 22 02:53:17.673641008 (564h45m38.166073489s) [noise=true , address=45.91.116.9:4262 , piece_id=AN2YEHJFWO3GV36HVYEUEBYWX7APL4GI2JGZVLTVMQRZSR74AENA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.677178652 (564h45m38.169611082s) [noise=true , address=45.91.116.10:4081 , piece_id=WDVMAL2L4I5MOUSECP6S4PXL3TK53SBRRLYAKWR2RXO7PWQLXJSA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.678967137 (564h45m38.171399573s) [noise=true , address=45.91.116.4:4937 , piece_id=HXJZJYSK3A2IAT3PTE5SA5UGEPCXATEJOKITCQ2D7CWTVPDGPEBA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.681878108 (564h45m38.174310536s) [noise=true , address=45.91.116.11:4118 , piece_id=5K5IYEDQU3PSFWG2IWNKWFOHWMG6BAQCPLIJLDHQQWEJYRNZ2GGA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.686890524 (564h45m38.179322957s) [noise=true , address=77.90.31.11:4555 , piece_id=BCTTNFWLDS3RA5XZRKJJIMRHGMXRFJMA2FUCB5EXSIVLRRZHESDA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.686940587 (564h45m38.179373029s) [noise=true , address=77.90.31.19:4841 , piece_id=Z52MG5CYIAUXMV32PHBPLYKGNCSJJSEGAK4DHHN3LJMYFHOJIL5A, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687424121 (564h45m38.179856556s) [noise=true , address=77.90.31.17:4770 , piece_id=OSZDK2HOGWIPYQHHY3AYZHTEPLJLKFJBGHVY5IMPEEQLL3BFN7PA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687535995 (564h45m38.179968424s) [noise=true , address=45.91.116.14:4190 , piece_id=LRN66OHJO5ST7F57Q66SGTW5T7Q2N4IK52TZXYWA232BYL4DNSCQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687981077 (564h45m38.180413511s) [noise=true , address=77.90.31.17:4771 , piece_id=MVI3VRDUA5K5O4N5UP2DP5OLCCJC44P4RAR4FLXJ5ZS77D27KXGA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.688037279 (564h45m38.180469715s) [noise=true , address=45.91.116.8:4045 , piece_id=VQRXTUUXBGHHINC26X75NGG3OAGRSJOJ3Y65EAXB7ENUQWFE6WRQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.690583072 (564h45m38.183015501s) [noise=true , address=45.91.116.6:4973 , piece_id=WUIM6UBQH7GEXD2BAVDDNYWF4GQFWWCL5MRXCKZ5UJS5W5634EBA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.697337949 (564h45m38.189770384s) [noise=true , address=45.91.116.5:4299 , piece_id=3IKQPV4BWIA2ON2GBOATUDK7LPZ3SVGIIV46QCSI5IGSO42SROPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.901510874 (564h45m38.393943323s) [noise=true , address=145.79.227.11:4732 , piece_id=KD4CBSFZ7RLL3FBKZ4NQKLS7PXFNQKXXGAOKKG5MERP423UJHOPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499213541 (564h46m6.991645996s) [noise=true , address=145.79.227.4:4186 , piece_id=2LFV4O5E5ZHYN24SEDLHHO334CDAW4AW7PAN322QXT3H2SRFSLPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499294035 (564h46m6.99172648s) [noise=true , address=145.79.227.16:4909 , piece_id=2F6FGN2RYDM5IUBZ5AXMQE7RYK3NKBKLCMGJOQ474KGWBMMHK2RQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499316904 (564h46m6.991749347s) [noise=true , address=145.79.227.18:4005 , piece_id=KXBNHWEZASVN6O4N76LRA5Z7EYGKSGJZWXX26FGJC5OZD5CVO6AA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499374361 (564h46m6.99180681s) [seg_pos={0 0}, upload=496844]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:53:46.611368399 (564h46m7.103800835s) [upload=496844]: Done uploading segments... Sep 22 02:53:46.611396228 (564h46m7.103828662s) [upload=496844]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.416021632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X....X......................................................] duration: 2.5683686s FAIL (2025-09-22 02:55:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.568395982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X....X.....................................................] duration: 2.657746281s FAIL (2025-09-22 02:54:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.657786332s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X.X.......X..X......X.X.....X.X...XX....X..........X..] duration: 2.669384438s FAIL (2025-09-22 02:54:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.669397174s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXX..X.X...XXXXXXX.X.X.X.XX.........X..X..X.X....X.....XX..] duration: 2.475165837s FAIL (2025-09-22 02:54:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:53:16.803730112 (564h45m37.296162545s) [upload=496844]: Starting upload Sep 22 02:53:16.803799753 (564h45m37.296232188s) [upload=496844]: Done starting upload Sep 22 02:53:16.803915386 (564h45m37.296347821s) [upload=496844]: Uploading segments... Sep 22 02:53:16.804081266 (564h45m37.296513698s) [upload=496844]: Got next segment. Inline: false Sep 22 02:53:16.804136100 (564h45m37.296568535s) [upload=496844]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:53:16.804142089 (564h45m37.296574532s) [upload=496844]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:53:16.924693065 (564h45m37.417125514s) [seg_pos={0 0}, upload=496844]: Begin upload segment... Sep 22 02:53:16.924761853 (564h45m37.417194292s) [seg_pos={0 0}, upload=496844]: Done begin upload segment. Sep 22 02:53:16.924794582 (564h45m37.417227034s) [noise=true , address=45.91.116.9:4262 , piece_id=AN2YEHJFWO3GV36HVYEUEBYWX7APL4GI2JGZVLTVMQRZSR74AENA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.925328296 (564h45m37.417760741s) [noise=true , address=77.90.31.17:4770 , piece_id=OSZDK2HOGWIPYQHHY3AYZHTEPLJLKFJBGHVY5IMPEEQLL3BFN7PA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.925670244 (564h45m37.418102687s) [noise=true , address=45.91.116.14:4190 , piece_id=LRN66OHJO5ST7F57Q66SGTW5T7Q2N4IK52TZXYWA232BYL4DNSCQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926006094 (564h45m37.418438536s) [noise=true , address=45.91.116.8:4045 , piece_id=VQRXTUUXBGHHINC26X75NGG3OAGRSJOJ3Y65EAXB7ENUQWFE6WRQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926206123 (564h45m37.418638568s) [noise=true , address=77.90.31.17:4771 , piece_id=MVI3VRDUA5K5O4N5UP2DP5OLCCJC44P4RAR4FLXJ5ZS77D27KXGA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.926834042 (564h45m37.419266491s) [noise=true , address=77.90.31.11:4555 , piece_id=BCTTNFWLDS3RA5XZRKJJIMRHGMXRFJMA2FUCB5EXSIVLRRZHESDA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927294956 (564h45m37.419727397s) [noise=true , address=45.91.116.6:4973 , piece_id=WUIM6UBQH7GEXD2BAVDDNYWF4GQFWWCL5MRXCKZ5UJS5W5634EBA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927573044 (564h45m37.420005486s) [noise=true , address=45.91.116.4:4937 , piece_id=HXJZJYSK3A2IAT3PTE5SA5UGEPCXATEJOKITCQ2D7CWTVPDGPEBA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.927864772 (564h45m37.420297214s) [noise=true , address=45.91.116.5:4299 , piece_id=3IKQPV4BWIA2ON2GBOATUDK7LPZ3SVGIIV46QCSI5IGSO42SROPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928013610 (564h45m37.420446133s) [noise=true , address=45.91.116.10:4081 , piece_id=WDVMAL2L4I5MOUSECP6S4PXL3TK53SBRRLYAKWR2RXO7PWQLXJSA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928149679 (564h45m37.420582123s) [noise=true , address=77.90.31.19:4841 , piece_id=Z52MG5CYIAUXMV32PHBPLYKGNCSJJSEGAK4DHHN3LJMYFHOJIL5A, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928546383 (564h45m37.420978825s) [noise=true , address=145.79.227.11:4732 , piece_id=KD4CBSFZ7RLL3FBKZ4NQKLS7PXFNQKXXGAOKKG5MERP423UJHOPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928891033 (564h45m37.421323475s) [noise=true , address=145.79.227.4:4186 , piece_id=2LFV4O5E5ZHYN24SEDLHHO334CDAW4AW7PAN322QXT3H2SRFSLPA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.928931827 (564h45m37.421364267s) [noise=true , address=145.79.227.16:4909 , piece_id=2F6FGN2RYDM5IUBZ5AXMQE7RYK3NKBKLCMGJOQ474KGWBMMHK2RQ, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.929444932 (564h45m37.421877365s) [noise=true , address=45.91.116.11:4118 , piece_id=5K5IYEDQU3PSFWG2IWNKWFOHWMG6BAQCPLIJLDHQQWEJYRNZ2GGA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:16.929986031 (564h45m37.422418464s) [noise=true , address=145.79.227.18:4005 , piece_id=KXBNHWEZASVN6O4N76LRA5Z7EYGKSGJZWXX26FGJC5OZD5CVO6AA, seg_pos={0 0}, upload=496844]: Uploading piece... Sep 22 02:53:17.532291405 (564h45m38.024723835s) [upload=496844]: Next returned nil segment Sep 22 02:53:17.532308544 (564h45m38.024740987s) [upload=496844]: Waiting for error group managing segments... Sep 22 02:53:17.673641008 (564h45m38.166073489s) [noise=true , address=45.91.116.9:4262 , piece_id=AN2YEHJFWO3GV36HVYEUEBYWX7APL4GI2JGZVLTVMQRZSR74AENA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.677178652 (564h45m38.169611082s) [noise=true , address=45.91.116.10:4081 , piece_id=WDVMAL2L4I5MOUSECP6S4PXL3TK53SBRRLYAKWR2RXO7PWQLXJSA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.678967137 (564h45m38.171399573s) [noise=true , address=45.91.116.4:4937 , piece_id=HXJZJYSK3A2IAT3PTE5SA5UGEPCXATEJOKITCQ2D7CWTVPDGPEBA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.681878108 (564h45m38.174310536s) [noise=true , address=45.91.116.11:4118 , piece_id=5K5IYEDQU3PSFWG2IWNKWFOHWMG6BAQCPLIJLDHQQWEJYRNZ2GGA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.686890524 (564h45m38.179322957s) [noise=true , address=77.90.31.11:4555 , piece_id=BCTTNFWLDS3RA5XZRKJJIMRHGMXRFJMA2FUCB5EXSIVLRRZHESDA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.686940587 (564h45m38.179373029s) [noise=true , address=77.90.31.19:4841 , piece_id=Z52MG5CYIAUXMV32PHBPLYKGNCSJJSEGAK4DHHN3LJMYFHOJIL5A, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687424121 (564h45m38.179856556s) [noise=true , address=77.90.31.17:4770 , piece_id=OSZDK2HOGWIPYQHHY3AYZHTEPLJLKFJBGHVY5IMPEEQLL3BFN7PA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687535995 (564h45m38.179968424s) [noise=true , address=45.91.116.14:4190 , piece_id=LRN66OHJO5ST7F57Q66SGTW5T7Q2N4IK52TZXYWA232BYL4DNSCQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.687981077 (564h45m38.180413511s) [noise=true , address=77.90.31.17:4771 , piece_id=MVI3VRDUA5K5O4N5UP2DP5OLCCJC44P4RAR4FLXJ5ZS77D27KXGA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.688037279 (564h45m38.180469715s) [noise=true , address=45.91.116.8:4045 , piece_id=VQRXTUUXBGHHINC26X75NGG3OAGRSJOJ3Y65EAXB7ENUQWFE6WRQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.690583072 (564h45m38.183015501s) [noise=true , address=45.91.116.6:4973 , piece_id=WUIM6UBQH7GEXD2BAVDDNYWF4GQFWWCL5MRXCKZ5UJS5W5634EBA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.697337949 (564h45m38.189770384s) [noise=true , address=45.91.116.5:4299 , piece_id=3IKQPV4BWIA2ON2GBOATUDK7LPZ3SVGIIV46QCSI5IGSO42SROPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: <nil> Sep 22 02:53:17.901510874 (564h45m38.393943323s) [noise=true , address=145.79.227.11:4732 , piece_id=KD4CBSFZ7RLL3FBKZ4NQKLS7PXFNQKXXGAOKKG5MERP423UJHOPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499213541 (564h46m6.991645996s) [noise=true , address=145.79.227.4:4186 , piece_id=2LFV4O5E5ZHYN24SEDLHHO334CDAW4AW7PAN322QXT3H2SRFSLPA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499294035 (564h46m6.99172648s) [noise=true , address=145.79.227.16:4909 , piece_id=2F6FGN2RYDM5IUBZ5AXMQE7RYK3NKBKLCMGJOQ474KGWBMMHK2RQ, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499316904 (564h46m6.991749347s) [noise=true , address=145.79.227.18:4005 , piece_id=KXBNHWEZASVN6O4N76LRA5Z7EYGKSGJZWXX26FGJC5OZD5CVO6AA, seg_pos={0 0}, upload=496844]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:53:46.499374361 (564h46m6.99180681s) [seg_pos={0 0}, upload=496844]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:53:46.611368399 (564h46m7.103800835s) [upload=496844]: Done uploading segments... Sep 22 02:53:46.611396228 (564h46m7.103828662s) [upload=496844]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.475250647s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)