Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 1m0.109059965s trace: https://jaeger.dev.storj.io/trace/abab3d9b6813e12 FAIL (2025-12-10 23:40:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2.049691206s) [upload=112678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:39:42.069345520 (128h11m22.049698165s) [upload=112678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:39:42.181592079 (128h11m22.16194472s) [seg_pos={0 0}, upload=112678]: Begin upload segment... Dec 10 23:39:42.181674938 (128h11m22.162027627s) [noise=true , address=77.90.31.12:4615 , piece_id=7N7FVFAVDAMXFP2CUFV5GER46AICERJ3LUP555YT46QZGHHIQURA, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.182813845 (128h11m22.163166485s) [noise=true , address=145.79.227.27:4704 , piece_id=JFHP4PIXAY3CMLQLIL2L64T7LZYVKY5E6547EU3NKF2SFYFC4ECQ, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.184073676 (128h11m22.164426325s) [noise=true , address=77.90.31.18:4811 , piece_id=J6QFG3TKXQGF2UQVROINIU46QHU56JM3KHO7H2YOFCEOEHNQDVDQ, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.186118150 (128h11m22.1664708s) [seg_pos={0 0}, upload=112678]: Done begin upload segment. Dec 10 23:39:42.186244342 (128h11m22.166596992s) [noise=true , address=77.90.31.17:4797 , piece_id=MTBE2EVGW3N2W6TPEDQQ467FAMEAM4DSIR3YMCDLP25WT4GDAK6Q, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.186318074 (128h11m22.16667073s) [noise=true , address=45.91.116.6:4006 , piece_id=OPKYLVQIM2QGMNS2RMS2AFTPA3DNBHQVRK2OB373NY44UUSMW4MQ, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.186934381 (128h11m22.167287031s) [noise=true , address=77.90.31.19:4845 , piece_id=6FMIXZLBV5RPHQJ2ZEPXO3ELMCKHCO6UVV5TZYAAIG64DHO72K6Q, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.186984986 (128h11m22.167337636s) [noise=true , address=45.91.116.5:4312 , piece_id=XEWL77RKMLHR7CII5WZB6WUY5TFJZVY6PATNKBHGJW2A2TKZ27GA, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.187427572 (128h11m22.167780232s) [noise=true , address=145.79.227.49:4357 , piece_id=T6UBCXGBHQ4E6Z2P255ONZYRKXQ7MGVHZZ3F6CSFJUK7IAXZ3KNQ, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.187626455 (128h11m22.167979106s) [noise=true , address=145.79.227.6:4268 , piece_id=BKSEOFSNLKTVMT44MENHGZVDVPPMX5IBFTYV7CY2TVI55OYIMTQA, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.187976077 (128h11m22.168328729s) [noise=true , address=5.231.73.5:4035 , piece_id=E4ABN3OUG5TBPVMX4J2SNTWSFYIXC2BH3CFO4U3GJ52AL4OFLV4A, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.188272483 (128h11m22.168625133s) [noise=true , address=193.149.164.143:4695 , piece_id=2O7V3UAUZWCTK7FJ25NWMG7KMWIXHUBK6EKNUOD7IAJYEEXAXO3A, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.188459321 (128h11m22.16881197s) [noise=true , address=45.91.116.13:4157 , piece_id=XJGPLZ3UGCSTWVKU3UB3CAQ7ENXYUBJFCQSY4W5KR3TZYHTHCQ2A, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.188911063 (128h11m22.169263712s) [noise=true , address=77.90.31.16:4764 , piece_id=B6NW6SAT7VNUF2NN7Y2UVDPCT2Y6W744DIGBIDQBFDKAGT3QBCSQ, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.188963480 (128h11m22.16931613s) [noise=true , address=145.79.227.44:4199 , piece_id=L7WK3MHYIUKRLTBU32DPATHZIGAJQ27HYE2IAQWXBGFZECIFAVBA, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.189487004 (128h11m22.169839648s) [noise=true , address=45.91.116.12:4240 , piece_id=H4JLDYVGUHRKNMVGEUBGMDVCI5FL54J7R4TPPLU4MT4PRRMKAWFA, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.189545533 (128h11m22.169898182s) [noise=true , address=45.91.116.8:4072 , piece_id=HJDZCGU43SXBDLO7ETREXVH6PIPK225BK3NP73LQLQIXU4VEANOQ, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.190867428 (128h11m22.171220079s) [noise=true , address=145.79.227.38:4983 , piece_id=OJFFJZ5V6GNYPVWHNOCRGWVU2FLCB7NRPCMK5DP73KAXH7RQBLTQ, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.191450576 (128h11m22.171803225s) [noise=true , address=64.49.13.140:4513 , piece_id=6B7MKH5DJDOPWJ5JH2XAK7RAD4CEI5XROV6W2NOCBES5S3DAB64Q, seg_pos={0 0}, upload=112678]: Uploading piece... Dec 10 23:39:42.735164699 (128h11m22.715517337s) [upload=112678]: Next returned nil segment Dec 10 23:39:42.735179894 (128h11m22.715532521s) [upload=112678]: Waiting for error group managing segments... Dec 10 23:39:42.883814214 (128h11m22.864166857s) [noise=true , address=45.91.116.6:4006 , piece_id=OPKYLVQIM2QGMNS2RMS2AFTPA3DNBHQVRK2OB373NY44UUSMW4MQ, seg_pos={0 0}, upload=112678]: Done uploading piece. err: <nil> Dec 10 23:39:42.884670320 (128h11m22.865022961s) [noise=true , address=145.79.227.27:4704 , piece_id=JFHP4PIXAY3CMLQLIL2L64T7LZYVKY5E6547EU3NKF2SFYFC4ECQ, seg_pos={0 0}, upload=112678]: Done uploading piece. err: <nil> Dec 10 23:39:42.885499034 (128h11m22.865851672s) [noise=true , address=45.91.116.13:4157 , piece_id=XJGPLZ3UGCSTWVKU3UB3CAQ7ENXYUBJFCQSY4W5KR3TZYHTHCQ2A, seg_pos={0 0}, upload=112678]: Done uploading piece. err: <nil> Dec 10 23:39:42.910288941 (128h11m22.890641578s) [noise=true , address=145.79.227.6:4268 , piece_id=BKSEOFSNLKTVMT44MENHGZVDVPPMX5IBFTYV7CY2TVI55OYIMTQA, seg_pos={0 0}, upload=112678]: Done uploading piece. err: <nil> Dec 10 23:39:42.916885985 (128h11m22.897238628s) [noise=true , address=45.91.116.5:4312 , piece_id=XEWL77RKMLHR7CII5WZB6WUY5TFJZVY6PATNKBHGJW2A2TKZ27GA, seg_pos={0 0}, upload=112678]: Done uploading piece. err: <nil> Dec 10 23:39:42.919280652 (128h11m22.899633289s) [noise=true , address=45.91.116.8:4072 , piece_id=HJDZCGU43SXBDLO7ETREXVH6PIPK225BK3NP73LQLQIXU4VEANOQ, seg_pos={0 0}, upload=112678]: Done uploading piece. err: <nil> Dec 10 23:39:42.932130099 (128h11m22.912482745s) [noise=true , address=45.91.116.12:4240 , piece_id=H4JLDYVGUHRKNMVGEUBGMDVCI5FL54J7R4TPPLU4MT4PRRMKAWFA, seg_pos={0 0}, upload=112678]: Done uploading piece. err: <nil> Dec 10 23:39:42.958976532 (128h11m22.939329181s) [noise=true , address=145.79.227.49:4357 , piece_id=T6UBCXGBHQ4E6Z2P255ONZYRKXQ7MGVHZZ3F6CSFJUK7IAXZ3KNQ, seg_pos={0 0}, upload=112678]: Done uploading piece. err: <nil> Dec 10 23:39:43.114284299 (128h11m23.094636937s) [noise=true , address=145.79.227.38:4983 , piece_id=OJFFJZ5V6GNYPVWHNOCRGWVU2FLCB7NRPCMK5DP73KAXH7RQBLTQ, seg_pos={0 0}, upload=112678]: Done uploading piece. err: <nil> Dec 10 23:39:43.118575490 (128h11m23.098928131s) [noise=true , address=5.231.73.5:4035 , piece_id=E4ABN3OUG5TBPVMX4J2SNTWSFYIXC2BH3CFO4U3GJ52AL4OFLV4A, seg_pos={0 0}, upload=112678]: Done uploading piece. err: <nil> Dec 10 23:39:43.579148669 (128h11m23.559501326s) [noise=true , address=193.149.164.143:4695 , piece_id=2O7V3UAUZWCTK7FJ25NWMG7KMWIXHUBK6EKNUOD7IAJYEEXAXO3A, seg_pos={0 0}, upload=112678]: Done uploading piece. err: <nil> Dec 10 23:39:43.580250068 (128h11m23.560602709s) [noise=true , address=64.49.13.140:4513 , piece_id=6B7MKH5DJDOPWJ5JH2XAK7RAD4CEI5XROV6W2NOCBES5S3DAB64Q, seg_pos={0 0}, upload=112678]: Done uploading piece. err: <nil> Dec 10 23:39:43.894991760 (128h11m23.875344404s) [noise=true , address=145.79.227.44:4199 , piece_id=L7WK3MHYIUKRLTBU32DPATHZIGAJQ27HYE2IAQWXBGFZECIFAVBA, seg_pos={0 0}, upload=112678]: Done uploading piece. err: <nil> Dec 10 23:40:41.776359086 (128h12m21.756711739s) [noise=true , address=77.90.31.18:4811 , piece_id=J6QFG3TKXQGF2UQVROINIU46QHU56JM3KHO7H2YOFCEOEHNQDVDQ, seg_pos={0 0}, upload=112678]: Done uploading piece. err: ecclient: upload failed (node:1AZxy9wnMgVu2PdFNtFrShYhWfF6PL4mgWumjqVk68voKsnMk6, address:77.90.31.18:4811): protocol: write tcp 139.59.144.22:52202->77.90.31.18:4811: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:41.776418599 (128h12m21.756771241s) [noise=true , address=77.90.31.19:4845 , piece_id=6FMIXZLBV5RPHQJ2ZEPXO3ELMCKHCO6UVV5TZYAAIG64DHO72K6Q, seg_pos={0 0}, upload=112678]: Done uploading piece. err: ecclient: upload failed (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, address:77.90.31.19:4845): protocol: write tcp 139.59.144.22:44796->77.90.31.19:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:41.776581250 (128h12m21.756933896s) [noise=true , address=77.90.31.17:4797 , piece_id=MTBE2EVGW3N2W6TPEDQQ467FAMEAM4DSIR3YMCDLP25WT4GDAK6Q, seg_pos={0 0}, upload=112678]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: write tcp 139.59.144.22:54130->77.90.31.17:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:41.776701727 (128h12m21.757054375s) [noise=true , address=77.90.31.16:4764 , piece_id=B6NW6SAT7VNUF2NN7Y2UVDPCT2Y6W744DIGBIDQBFDKAGT3QBCSQ, seg_pos={0 0}, upload=112678]: Done uploading piece. err: ecclient: upload failed (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, address:77.90.31.16:4764): protocol: write tcp 139.59.144.22:54234->77.90.31.16:4764: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:41.776972512 (128h12m21.757325159s) [noise=true , address=77.90.31.12:4615 , piece_id=7N7FVFAVDAMXFP2CUFV5GER46AICERJ3LUP555YT46QZGHHIQURA, seg_pos={0 0}, upload=112678]: Done uploading piece. err: ecclient: upload failed (node:1xaory9AHtW4ra15wmL2CagPKKJU2u9keFo1ntBMy8AzjpkmwR, address:77.90.31.12:4615): protocol: write tcp 139.59.144.22:56210->77.90.31.12:4615: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:41.777024367 (128h12m21.757377006s) [seg_pos={0 0}, upload=112678]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:40:41.882326362 (128h12m21.862679004s) [upload=112678]: Done uploading segments... Dec 10 23:40:41.882371820 (128h12m21.862724468s) [upload=112678]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.109086206s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................................XXX..................] duration: 1.597389696s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:40:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.597423545s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................................XXX...................] duration: 2.801835891s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:40:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.801860958s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.....X....] duration: 2.518061651s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:40:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.518091806s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................................XXX..................] duration: 3.360764857s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:39:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.360792119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 1m0.112579087s trace: https://jaeger.dev.storj.io/trace/925444167d3f530 FAIL (2025-12-10 23:39:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== nt. Dec 10 23:38:41.943397183 (128h10m21.923749819s) [noise=true , address=45.91.116.6:4995 , piece_id=5X6JOTNUA6WAU35GOYWUTLRMATNXQMDAHZNGX7JDQC7DMMZ3MZIA, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.944610988 (128h10m21.924963633s) [noise=true , address=5.231.73.4:4028 , piece_id=NOERTQAW6HI7MERGAUYFJXFJEX7SYSKW5L7DIS6IHQGH5NYGQ6SQ, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.944726596 (128h10m21.925079235s) [noise=true , address=77.90.31.18:4826 , piece_id=DIJEMSWGIPNUPE3MRRSMAOLV3CGRQVEQMGCGG72ZZ225BYOYPD2Q, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.945357379 (128h10m21.925710025s) [noise=true , address=145.79.227.23:4548 , piece_id=6743WG7JWL7VD4U4C2KYXIBHFQ4YKLQ63J5VDE6HTNCY3YCJBHEA, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.945375205 (128h10m21.92572784s) [noise=true , address=145.79.227.24:4613 , piece_id=L56CSY53V47MFESUBKP4DE42PCTYISLUGIRE75WZOGRZLG5M3W5A, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.945845035 (128h10m21.926197678s) [noise=true , address=145.79.227.50:4376 , piece_id=JNVLBQADRMLVF3MBY75GUOTAE2M6EZZQZLZSHSYNFBQJ6C6HOAEA, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.945871759 (128h10m21.926224406s) [noise=true , address=45.91.116.5:4310 , piece_id=QKKXVO7VS5GUAIQIF4MZ7RMKR3ZD27OCJDOSDSX23XTNHPPQOHZQ, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.946357361 (128h10m21.926710007s) [noise=true , address=77.90.31.10:4540 , piece_id=PKSERNUFME3WYCS7YON3Y4JAC4ES4I4KZACZS2OVEKLUN4FXD7PQ, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.946375974 (128h10m21.926728623s) [noise=true , address=77.90.31.16:4745 , piece_id=LTAL537QFBEVWS7JRPW4I24YYDLT26ZLRDY2F3HLV32SYMOT75UQ, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.946835999 (128h10m21.927188647s) [noise=true , address=77.90.31.19:4868 , piece_id=Q5DUUCI4MUKPDSI5GKELAYAVEAIITONJTOSTMEUS4OOK3SCCGO2A, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.946874167 (128h10m21.927226806s) [noise=true , address=45.91.116.11:4131 , piece_id=4AHOC4VD6XNVUH2LDY3RPRTAKKU7JX6KMJMAFLEVRXK5GXJYKEKA, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.947283994 (128h10m21.927636637s) [noise=true , address=145.79.227.49:4347 , piece_id=XEYKQ2PQ2ZLEENKL5NQWYLQRZCPRNFUBTBTXF3XOTKGPPXWKSYOA, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.947399328 (128h10m21.927751968s) [noise=true , address=77.90.31.16:4764 , piece_id=BGAUYIKXTSHSBUV42U3W27VQK5O6Y53WPKYPC67AQUXYD6NWJAOQ, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.947910487 (128h10m21.928263124s) [noise=true , address=45.91.116.10:4107 , piece_id=A7QAL64EQM7N7K4BGXAYB3KA2XRVWGLJMOJNWMZ6CQJ2L7NYEFCQ, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.948367527 (128h10m21.928720163s) [noise=true , address=77.90.31.21:4887 , piece_id=GG3MMRUMPK47E3YGTJWVH2WB3V7XESVS3IULPCPLHTGABRWBZXXQ, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.948794167 (128h10m21.92914681s) [noise=true , address=145.79.227.16:4929 , piece_id=5J3RQQFWI3DI5FX3VSUWZ4T3UNPSWQ55CTPNQIOVFV6YQWG7X23A, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.949259700 (128h10m21.929612342s) [noise=true , address=145.79.227.25:4634 , piece_id=IK5FEKOT7QN7G6DW746WLK3HXMHRPOS43OXEKCCHQJFDTBTUQ4NQ, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:41.950380580 (128h10m21.930733209s) [noise=true , address=77.90.31.11:4585 , piece_id=YEFAU4S2NKCB6WP3UCSSPGL7VMSV4UHAAN7A3QEZIN4I5MN6MA3Q, seg_pos={0 0}, upload=112663]: Uploading piece... Dec 10 23:38:42.502608850 (128h10m22.482961487s) [upload=112663]: Next returned nil segment Dec 10 23:38:42.502625331 (128h10m22.482977957s) [upload=112663]: Waiting for error group managing segments... Dec 10 23:38:42.652202336 (128h10m22.632554971s) [noise=true , address=145.79.227.50:4376 , piece_id=JNVLBQADRMLVF3MBY75GUOTAE2M6EZZQZLZSHSYNFBQJ6C6HOAEA, seg_pos={0 0}, upload=112663]: Done uploading piece. err: <nil> Dec 10 23:38:42.652774286 (128h10m22.633126922s) [noise=true , address=145.79.227.25:4634 , piece_id=IK5FEKOT7QN7G6DW746WLK3HXMHRPOS43OXEKCCHQJFDTBTUQ4NQ, seg_pos={0 0}, upload=112663]: Done uploading piece. err: <nil> Dec 10 23:38:42.667714629 (128h10m22.648067266s) [noise=true , address=45.91.116.10:4107 , piece_id=A7QAL64EQM7N7K4BGXAYB3KA2XRVWGLJMOJNWMZ6CQJ2L7NYEFCQ, seg_pos={0 0}, upload=112663]: Done uploading piece. err: <nil> Dec 10 23:38:42.668559648 (128h10m22.648912285s) [noise=true , address=145.79.227.24:4613 , piece_id=L56CSY53V47MFESUBKP4DE42PCTYISLUGIRE75WZOGRZLG5M3W5A, seg_pos={0 0}, upload=112663]: Done uploading piece. err: <nil> Dec 10 23:38:42.671942317 (128h10m22.652294953s) [noise=true , address=145.79.227.49:4347 , piece_id=XEYKQ2PQ2ZLEENKL5NQWYLQRZCPRNFUBTBTXF3XOTKGPPXWKSYOA, seg_pos={0 0}, upload=112663]: Done uploading piece. err: <nil> Dec 10 23:38:42.676787736 (128h10m22.657140373s) [noise=true , address=45.91.116.5:4310 , piece_id=QKKXVO7VS5GUAIQIF4MZ7RMKR3ZD27OCJDOSDSX23XTNHPPQOHZQ, seg_pos={0 0}, upload=112663]: Done uploading piece. err: <nil> Dec 10 23:38:42.677558528 (128h10m22.657911164s) [noise=true , address=45.91.116.6:4995 , piece_id=5X6JOTNUA6WAU35GOYWUTLRMATNXQMDAHZNGX7JDQC7DMMZ3MZIA, seg_pos={0 0}, upload=112663]: Done uploading piece. err: <nil> Dec 10 23:38:42.680897074 (128h10m22.661249728s) [noise=true , address=145.79.227.23:4548 , piece_id=6743WG7JWL7VD4U4C2KYXIBHFQ4YKLQ63J5VDE6HTNCY3YCJBHEA, seg_pos={0 0}, upload=112663]: Done uploading piece. err: <nil> Dec 10 23:38:42.681827248 (128h10m22.662179885s) [noise=true , address=45.91.116.11:4131 , piece_id=4AHOC4VD6XNVUH2LDY3RPRTAKKU7JX6KMJMAFLEVRXK5GXJYKEKA, seg_pos={0 0}, upload=112663]: Done uploading piece. err: <nil> Dec 10 23:38:42.873847774 (128h10m22.854200409s) [noise=true , address=5.231.73.4:4028 , piece_id=NOERTQAW6HI7MERGAUYFJXFJEX7SYSKW5L7DIS6IHQGH5NYGQ6SQ, seg_pos={0 0}, upload=112663]: Done uploading piece. err: <nil> Dec 10 23:38:43.672370566 (128h10m23.652723208s) [noise=true , address=145.79.227.16:4929 , piece_id=5J3RQQFWI3DI5FX3VSUWZ4T3UNPSWQ55CTPNQIOVFV6YQWG7X23A, seg_pos={0 0}, upload=112663]: Done uploading piece. err: <nil> Dec 10 23:39:41.530527910 (128h11m21.510880556s) [noise=true , address=77.90.31.16:4764 , piece_id=BGAUYIKXTSHSBUV42U3W27VQK5O6Y53WPKYPC67AQUXYD6NWJAOQ, seg_pos={0 0}, upload=112663]: Done uploading piece. err: ecclient: upload failed (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, address:77.90.31.16:4764): protocol: write tcp 139.59.144.22:46936->77.90.31.16:4764: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:41.530753279 (128h11m21.511105926s) [noise=true , address=77.90.31.10:4540 , piece_id=PKSERNUFME3WYCS7YON3Y4JAC4ES4I4KZACZS2OVEKLUN4FXD7PQ, seg_pos={0 0}, upload=112663]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: write tcp 139.59.144.22:46512->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:41.530764663 (128h11m21.511117313s) [noise=true , address=77.90.31.18:4826 , piece_id=DIJEMSWGIPNUPE3MRRSMAOLV3CGRQVEQMGCGG72ZZ225BYOYPD2Q, seg_pos={0 0}, upload=112663]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: write tcp 139.59.144.22:57950->77.90.31.18:4826: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:41.530943417 (128h11m21.511296064s) [noise=true , address=77.90.31.11:4585 , piece_id=YEFAU4S2NKCB6WP3UCSSPGL7VMSV4UHAAN7A3QEZIN4I5MN6MA3Q, seg_pos={0 0}, upload=112663]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: rpc: dial tcp 77.90.31.11:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:41.530961365 (128h11m21.511314015s) [noise=true , address=77.90.31.21:4887 , piece_id=GG3MMRUMPK47E3YGTJWVH2WB3V7XESVS3IULPCPLHTGABRWBZXXQ, seg_pos={0 0}, upload=112663]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: write tcp 139.59.144.22:33924->77.90.31.21:4887: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:41.531169432 (128h11m21.511522082s) [noise=true , address=77.90.31.19:4868 , piece_id=Q5DUUCI4MUKPDSI5GKELAYAVEAIITONJTOSTMEUS4OOK3SCCGO2A, seg_pos={0 0}, upload=112663]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:41.531369003 (128h11m21.511721626s) [noise=true , address=77.90.31.16:4745 , piece_id=LTAL537QFBEVWS7JRPW4I24YYDLT26ZLRDY2F3HLV32SYMOT75UQ, seg_pos={0 0}, upload=112663]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: rpc: dial tcp 77.90.31.16:4745: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:41.531423833 (128h11m21.511776471s) [seg_pos={0 0}, upload=112663]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:39:41.640793436 (128h11m21.621146072s) [upload=112663]: Done uploading segments... Dec 10 23:39:41.640828173 (128h11m21.621180812s) [upload=112663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.112609221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................................XXX.................] duration: 1.673352936s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:39:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.67339517s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.....X...] duration: 2.537282047s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:39:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.537377517s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................................XXX.................] duration: 2.761999305s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:39:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.762025063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.....X..] duration: 2.783115863s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:39:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.783138596s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................................XXX................] duration: 1.586309838s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:39:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.586345612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 1m0.115886445s trace: https://jaeger.dev.storj.io/trace/183f9b4866d29723 FAIL (2025-12-10 23:38:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 4:4958 , piece_id=AGH4R4IOFP4TUEEXP6RQHOYE7NYUVJEJATPYPXIC656KVYY7CX7A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.106237966 (128h9m13.086590845s) [noise=true , address=145.79.227.23:4561 , piece_id=NOUCJYHP7MKP36R55SVT7I2INJY3OPVP4EVLMGLMI5WFUGQ3FP6A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.106332383 (128h9m13.086685029s) [noise=true , address=77.90.31.13:4660 , piece_id=B4CIA3CAWP7VP5PMLP5BUVLB7LRKXSJIKYB27XPUH4CM2DAETUBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.106919275 (128h9m13.087271959s) [noise=true , address=64.49.13.139:4480 , piece_id=W3S7CG2NZIKF7HIWNFH6WN543BZ6TZHP7MFKJL42J2MUGY4QSSUA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.106995709 (128h9m13.087348347s) [noise=true , address=145.79.227.34:4917 , piece_id=UKVVVL7WDXCHEDMTYIPDXL2ILXAB6UFFU2R4DFTTCNWMDOVZZK3A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.107400534 (128h9m13.087753177s) [noise=true , address=77.90.31.14:4684 , piece_id=ZDEP5YEPQS3DGQA5KUEFM37NMWBPPUBNPLC4AF4MK7MLCAE22ADA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.107758839 (128h9m13.088111484s) [noise=true , address=45.91.116.9:4276 , piece_id=AHQCG5Q6RDDFAN54ELWEZ7ZUQOVUHQIFOQJU62PSG22DYKJBLF3A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.108127312 (128h9m13.08847996s) [noise=true , address=45.91.116.12:4248 , piece_id=LNVSQQJQIP35DKC7ZQTUE5VDDCQXUKDO75VHTPAMPB4BBMNE3CMQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.108478579 (128h9m13.08883122s) [noise=true , address=77.90.31.10:4541 , piece_id=YVCQRFTQ3NSC5D5Y3P5TEOS7EP6LRJMCQ3W25CSMDCEEADYSRQVA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.108799107 (128h9m13.08915175s) [noise=true , address=145.79.227.36:4526 , piece_id=X3KG2T3NNJFGKAYGFFYRSM6HNTEGMHLEHWRLJIJLUROKDY3DCX6A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.109252793 (128h9m13.089605475s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVTOMORTLFH3RL2EE7R2E4ODIH5P2CDW3MI7GMT6ITZBYHWT7EQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.110497482 (128h9m13.090850124s) [noise=true , address=45.91.116.12:4239 , piece_id=HO7RMWZJSDY2LX5MA2PJJRW3CLD7GPZUNDMLCKPWT3U6WOCATEFQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.110911845 (128h9m13.091264481s) [noise=true , address=45.91.116.8:4049 , piece_id=6JIT37MGHIXRK522EU2C6Q562UBFHN6R4VI7BEGF3HTCXWR5LJMQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.112431972 (128h9m13.092784611s) [noise=true , address=77.90.31.15:4715 , piece_id=LQ3A7J527JKOTNM4KF6EDIT5SKAF6S74UGS4244UXCJOXLNOJ46Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.113963098 (128h9m13.094315745s) [noise=true , address=77.90.31.11:4570 , piece_id=RWU4F6J2NIHH6NKFSUCTY3IKORE6C26BKWG5EO5ZYGZRSHMCJSIQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.115197447 (128h9m13.095550089s) [noise=true , address=145.79.227.7:4312 , piece_id=SCT3POKZIOBWCZK6JMURMDIFBEJXTMKGWKRBMQGN2WTWOVBZMKWA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.116137273 (128h9m13.096489919s) [noise=true , address=77.90.31.21:4906 , piece_id=5QDZ26RZB644ZIOVF5DKOYGACFBIBMTLU3ZJMHQHCQBRNQCGQ2KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.116820099 (128h9m13.097172741s) [noise=true , address=77.90.31.18:4811 , piece_id=KGVZ63FH7XVEKOAILCJB5QHZ5A2NOHXFJTFTVCU6TPU2KPD2CWQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:37:33.665479377 (128h9m13.645832017s) [upload=112648]: Next returned nil segment Dec 10 23:37:33.665502131 (128h9m13.645854778s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:37:33.816244617 (128h9m13.796597262s) [noise=true , address=145.79.227.23:4561 , piece_id=NOUCJYHP7MKP36R55SVT7I2INJY3OPVP4EVLMGLMI5WFUGQ3FP6A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:37:33.823868476 (128h9m13.804221125s) [noise=true , address=45.91.116.12:4248 , piece_id=LNVSQQJQIP35DKC7ZQTUE5VDDCQXUKDO75VHTPAMPB4BBMNE3CMQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:37:33.824912151 (128h9m13.805264799s) [noise=true , address=45.91.116.4:4958 , piece_id=AGH4R4IOFP4TUEEXP6RQHOYE7NYUVJEJATPYPXIC656KVYY7CX7A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:37:33.828153452 (128h9m13.808506096s) [noise=true , address=45.91.116.12:4239 , piece_id=HO7RMWZJSDY2LX5MA2PJJRW3CLD7GPZUNDMLCKPWT3U6WOCATEFQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:37:33.835441318 (128h9m13.815793966s) [noise=true , address=145.79.227.34:4917 , piece_id=UKVVVL7WDXCHEDMTYIPDXL2ILXAB6UFFU2R4DFTTCNWMDOVZZK3A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:37:33.838851096 (128h9m13.819203734s) [noise=true , address=45.91.116.9:4276 , piece_id=AHQCG5Q6RDDFAN54ELWEZ7ZUQOVUHQIFOQJU62PSG22DYKJBLF3A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:37:33.838918524 (128h9m13.819271169s) [noise=true , address=45.91.116.8:4049 , piece_id=6JIT37MGHIXRK522EU2C6Q562UBFHN6R4VI7BEGF3HTCXWR5LJMQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:37:33.840206307 (128h9m13.820558963s) [noise=true , address=145.79.227.36:4526 , piece_id=X3KG2T3NNJFGKAYGFFYRSM6HNTEGMHLEHWRLJIJLUROKDY3DCX6A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:37:33.860990606 (128h9m13.841343247s) [noise=true , address=145.79.227.7:4312 , piece_id=SCT3POKZIOBWCZK6JMURMDIFBEJXTMKGWKRBMQGN2WTWOVBZMKWA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:37:34.349901269 (128h9m14.330253914s) [noise=true , address=64.49.13.139:4480 , piece_id=W3S7CG2NZIKF7HIWNFH6WN543BZ6TZHP7MFKJL42J2MUGY4QSSUA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:37:34.764002291 (128h9m14.744354943s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVTOMORTLFH3RL2EE7R2E4ODIH5P2CDW3MI7GMT6ITZBYHWT7EQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:38:32.670292318 (128h10m12.650644966s) [noise=true , address=77.90.31.11:4570 , piece_id=RWU4F6J2NIHH6NKFSUCTY3IKORE6C26BKWG5EO5ZYGZRSHMCJSIQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: write tcp 139.59.144.22:51898->77.90.31.11:4570: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:32.670509566 (128h10m12.650862214s) [noise=true , address=77.90.31.18:4811 , piece_id=KGVZ63FH7XVEKOAILCJB5QHZ5A2NOHXFJTFTVCU6TPU2KPD2CWQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1AZxy9wnMgVu2PdFNtFrShYhWfF6PL4mgWumjqVk68voKsnMk6, address:77.90.31.18:4811): protocol: write tcp 139.59.144.22:43420->77.90.31.18:4811: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:32.670706245 (128h10m12.651058891s) [noise=true , address=77.90.31.21:4906 , piece_id=5QDZ26RZB644ZIOVF5DKOYGACFBIBMTLU3ZJMHQHCQBRNQCGQ2KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: write tcp 139.59.144.22:42672->77.90.31.21:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:32.670885004 (128h10m12.65123765s) [noise=true , address=77.90.31.14:4684 , piece_id=ZDEP5YEPQS3DGQA5KUEFM37NMWBPPUBNPLC4AF4MK7MLCAE22ADA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12a7wpA6YchhigZCKC68YJzkBKwbqg1R1gBdfN8GdvrVregrK4m, address:77.90.31.14:4684): protocol: write tcp 139.59.144.22:35202->77.90.31.14:4684: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:32.670944569 (128h10m12.651297211s) [noise=true , address=77.90.31.10:4541 , piece_id=YVCQRFTQ3NSC5D5Y3P5TEOS7EP6LRJMCQ3W25CSMDCEEADYSRQVA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW, address:77.90.31.10:4541): protocol: write tcp 139.59.144.22:57528->77.90.31.10:4541: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:32.671031352 (128h10m12.651383975s) [noise=true , address=77.90.31.13:4660 , piece_id=B4CIA3CAWP7VP5PMLP5BUVLB7LRKXSJIKYB27XPUH4CM2DAETUBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:50888->77.90.31.13:4660: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:32.671187974 (128h10m12.651540602s) [noise=true , address=77.90.31.15:4715 , piece_id=LQ3A7J527JKOTNM4KF6EDIT5SKAF6S74UGS4244UXCJOXLNOJ46Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 139.59.144.22:51170->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:32.671249929 (128h10m12.651602556s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:38:32.783227529 (128h10m12.763580167s) [upload=112648]: Done uploading segments... Dec 10 23:38:32.783261561 (128h10m12.763614203s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.116001697s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................................XXX...............] duration: 1.662562651s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:38:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.662657728s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................XXX................] duration: 2.644727048s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:38:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.644941398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.....X.] duration: 2.736743939s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:38:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.736794265s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................................XXX..............] duration: 1.67674383s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:37:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.67677325s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.....X] duration: 30.001601458s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:37:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001629902s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 1m0.116953814s trace: https://jaeger.dev.storj.io/trace/cd3d90abd05dbb3 FAIL (2025-12-10 23:37:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4207162s) [upload=112633]: Uploading segments... Dec 10 23:36:28.662240907 (128h8m8.642593554s) [upload=112633]: Got next segment. Inline: false Dec 10 23:36:28.662340749 (128h8m8.642693391s) [upload=112633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:36:28.662348585 (128h8m8.642701228s) [upload=112633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:36:28.771910605 (128h8m8.752263241s) [seg_pos={0 0}, upload=112633]: Begin upload segment... Dec 10 23:36:28.771962295 (128h8m8.752314932s) [seg_pos={0 0}, upload=112633]: Done begin upload segment. Dec 10 23:36:28.771992842 (128h8m8.752345481s) [noise=true , address=45.91.116.12:4241 , piece_id=7PGQ4DHTG25YFGUPPJKWYQKCJZHXUU7SE47APANPPS5A7WJQPEYA, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.772006215 (128h8m8.752358851s) [noise=true , address=77.90.31.17:4784 , piece_id=AXVVH42RX6SZD7RAOD3GEAADFDW3IEVSZCPYQ44TIJ6Q77IMUMYQ, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.772415182 (128h8m8.75276781s) [noise=true , address=145.79.227.34:4911 , piece_id=OBH6A5YTBVJUADSGN64FV4E5MTNRVUHQDEPTJRPKN2CPNIXXX4VA, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.772823425 (128h8m8.753176072s) [noise=true , address=145.79.227.4:4211 , piece_id=2KOYVWQSN5B2G2VKY5WQNRRBEXN4MLL365O645I3GE3X6VG3ZSHQ, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.773356781 (128h8m8.753709429s) [noise=true , address=77.90.31.11:4586 , piece_id=MKTLES5MAQWPAUQDL7A36DNNRYGLAIAXDPWLBQ45ZX33J4BUDKNA, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.773844144 (128h8m8.75419679s) [noise=true , address=145.79.227.46:4619 , piece_id=PQYYNRIFHXPMHVMSWBK7WZIV5TB6B7BYUBNHJHUGOSTEBQCDQQLA, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.774346471 (128h8m8.754699117s) [noise=true , address=5.231.73.5:4007 , piece_id=W2T7JGKCHPY4CB2LCPEQQAYX6WBKYNDBET3G5CFRIXGXBIIITBFQ, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.774855237 (128h8m8.755207883s) [noise=true , address=77.90.31.10:4530 , piece_id=UEARSWMAYDVQHPJMEPMCUSRUZKQRG7AMIIB24YUBNYNDMZXCOX2Q, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.775303628 (128h8m8.755656273s) [noise=true , address=145.79.227.10:4418 , piece_id=GW3FOYSGZM34Q7GJVXC3TFMFVQLKN2CJMYZO34HMP6YAUW2RU66Q, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.775799370 (128h8m8.756152018s) [noise=true , address=77.90.31.15:4723 , piece_id=OIADNKVXTYEMSM7PFMF7XLHTK4TABNXU2ILGEWORMNK6X4D55ROA, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.776359254 (128h8m8.756711897s) [noise=true , address=145.79.227.25:4631 , piece_id=4UC67HF76HUD63O5DGGZY2ZNDWACUVB6BH75XZ22YGVWFJVRZ4YA, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.776967668 (128h8m8.757320315s) [noise=true , address=45.91.116.13:4176 , piece_id=6T244PYXCV2FDQSIN5UKWQDIZFUDAVOJUNTZNM2UBY2SW5NRKIVQ, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.777680896 (128h8m8.758033544s) [noise=true , address=145.79.227.29:4947 , piece_id=QFTZ3BTN3HM4NRSY72NJIUZRYVMMQTMQUK5ZH7PI2CVCZ57PEY2Q, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.779243419 (128h8m8.759596049s) [noise=true , address=45.91.116.9:4279 , piece_id=TURHZZR5HC75H5I5QYOFE4CCSUGUVJBBD7ZVQEQJZIPIFRASPZMQ, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.779440765 (128h8m8.759793412s) [noise=true , address=77.90.31.12:4621 , piece_id=HSFLOE4AX3TXN2VKS4NG7OSE54LBTJMNIVI3YLNNV63ECIPQUECA, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.781386805 (128h8m8.761739454s) [noise=true , address=145.79.227.49:4356 , piece_id=R2QMFBZ5UZ2JONNAE2L2ZGNWLVESTMHDJXPMNVVYOGMIXJNXIDAA, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.783198739 (128h8m8.763551383s) [noise=true , address=45.91.116.10:4111 , piece_id=QOHCXNDFCZAY2PGXBLJW5PBOKIIT26T2UAWCZH2VTNXXN6X42TOA, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:28.784726369 (128h8m8.765079012s) [noise=true , address=5.231.73.2:4020 , piece_id=PJCVVIRXIKIZGZJXW73IEZAQ5JDWIHEMVWBNR4YS6KLPM4IISJ7Q, seg_pos={0 0}, upload=112633]: Uploading piece... Dec 10 23:36:29.341224229 (128h8m9.321576866s) [upload=112633]: Next returned nil segment Dec 10 23:36:29.341241487 (128h8m9.321594124s) [upload=112633]: Waiting for error group managing segments... Dec 10 23:36:29.480211033 (128h8m9.460563669s) [noise=true , address=145.79.227.10:4418 , piece_id=GW3FOYSGZM34Q7GJVXC3TFMFVQLKN2CJMYZO34HMP6YAUW2RU66Q, seg_pos={0 0}, upload=112633]: Done uploading piece. err: <nil> Dec 10 23:36:29.482956419 (128h8m9.463309055s) [noise=true , address=45.91.116.9:4279 , piece_id=TURHZZR5HC75H5I5QYOFE4CCSUGUVJBBD7ZVQEQJZIPIFRASPZMQ, seg_pos={0 0}, upload=112633]: Done uploading piece. err: <nil> Dec 10 23:36:29.489311825 (128h8m9.469664462s) [noise=true , address=45.91.116.13:4176 , piece_id=6T244PYXCV2FDQSIN5UKWQDIZFUDAVOJUNTZNM2UBY2SW5NRKIVQ, seg_pos={0 0}, upload=112633]: Done uploading piece. err: <nil> Dec 10 23:36:29.489545853 (128h8m9.469898495s) [noise=true , address=145.79.227.46:4619 , piece_id=PQYYNRIFHXPMHVMSWBK7WZIV5TB6B7BYUBNHJHUGOSTEBQCDQQLA, seg_pos={0 0}, upload=112633]: Done uploading piece. err: <nil> Dec 10 23:36:29.497618526 (128h8m9.477971162s) [noise=true , address=145.79.227.4:4211 , piece_id=2KOYVWQSN5B2G2VKY5WQNRRBEXN4MLL365O645I3GE3X6VG3ZSHQ, seg_pos={0 0}, upload=112633]: Done uploading piece. err: <nil> Dec 10 23:36:29.500020810 (128h8m9.480373452s) [noise=true , address=45.91.116.12:4241 , piece_id=7PGQ4DHTG25YFGUPPJKWYQKCJZHXUU7SE47APANPPS5A7WJQPEYA, seg_pos={0 0}, upload=112633]: Done uploading piece. err: <nil> Dec 10 23:36:29.508016523 (128h8m9.48836916s) [noise=true , address=145.79.227.34:4911 , piece_id=OBH6A5YTBVJUADSGN64FV4E5MTNRVUHQDEPTJRPKN2CPNIXXX4VA, seg_pos={0 0}, upload=112633]: Done uploading piece. err: <nil> Dec 10 23:36:29.511245988 (128h8m9.491598626s) [noise=true , address=145.79.227.49:4356 , piece_id=R2QMFBZ5UZ2JONNAE2L2ZGNWLVESTMHDJXPMNVVYOGMIXJNXIDAA, seg_pos={0 0}, upload=112633]: Done uploading piece. err: <nil> Dec 10 23:36:29.515092764 (128h8m9.4954454s) [noise=true , address=45.91.116.10:4111 , piece_id=QOHCXNDFCZAY2PGXBLJW5PBOKIIT26T2UAWCZH2VTNXXN6X42TOA, seg_pos={0 0}, upload=112633]: Done uploading piece. err: <nil> Dec 10 23:36:29.518373042 (128h8m9.498725682s) [noise=true , address=145.79.227.25:4631 , piece_id=4UC67HF76HUD63O5DGGZY2ZNDWACUVB6BH75XZ22YGVWFJVRZ4YA, seg_pos={0 0}, upload=112633]: Done uploading piece. err: <nil> Dec 10 23:36:29.518543847 (128h8m9.498896488s) [noise=true , address=145.79.227.29:4947 , piece_id=QFTZ3BTN3HM4NRSY72NJIUZRYVMMQTMQUK5ZH7PI2CVCZ57PEY2Q, seg_pos={0 0}, upload=112633]: Done uploading piece. err: <nil> Dec 10 23:36:29.645575924 (128h8m9.625928561s) [noise=true , address=5.231.73.5:4007 , piece_id=W2T7JGKCHPY4CB2LCPEQQAYX6WBKYNDBET3G5CFRIXGXBIIITBFQ, seg_pos={0 0}, upload=112633]: Done uploading piece. err: <nil> Dec 10 23:36:30.080129180 (128h8m10.060481827s) [noise=true , address=5.231.73.2:4020 , piece_id=PJCVVIRXIKIZGZJXW73IEZAQ5JDWIHEMVWBNR4YS6KLPM4IISJ7Q, seg_pos={0 0}, upload=112633]: Done uploading piece. err: <nil> Dec 10 23:37:28.368853018 (128h9m8.349205664s) [noise=true , address=77.90.31.15:4723 , piece_id=OIADNKVXTYEMSM7PFMF7XLHTK4TABNXU2ILGEWORMNK6X4D55ROA, seg_pos={0 0}, upload=112633]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: write tcp 139.59.144.22:42976->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:28.369171023 (128h9m8.349523663s) [noise=true , address=77.90.31.11:4586 , piece_id=MKTLES5MAQWPAUQDL7A36DNNRYGLAIAXDPWLBQ45ZX33J4BUDKNA, seg_pos={0 0}, upload=112633]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:55720->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:28.369607034 (128h9m8.349959677s) [noise=true , address=77.90.31.17:4784 , piece_id=AXVVH42RX6SZD7RAOD3GEAADFDW3IEVSZCPYQ44TIJ6Q77IMUMYQ, seg_pos={0 0}, upload=112633]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: write tcp 139.59.144.22:60726->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:28.369792531 (128h9m8.35014518s) [noise=true , address=77.90.31.12:4621 , piece_id=HSFLOE4AX3TXN2VKS4NG7OSE54LBTJMNIVI3YLNNV63ECIPQUECA, seg_pos={0 0}, upload=112633]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: write tcp 139.59.144.22:56068->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:28.370218966 (128h9m8.350571603s) [noise=true , address=77.90.31.10:4530 , piece_id=UEARSWMAYDVQHPJMEPMCUSRUZKQRG7AMIIB24YUBNYNDMZXCOX2Q, seg_pos={0 0}, upload=112633]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:37:28.370248550 (128h9m8.350601175s) [seg_pos={0 0}, upload=112633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:37:28.483425634 (128h9m8.463778271s) [upload=112633]: Done uploading segments... Dec 10 23:37:28.483461713 (128h9m8.463814352s) [upload=112633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.116969354s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........................................XXX...............] duration: 2.681334845s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:37:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.681359168s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................................XXX.............] duration: 1.660095823s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:37:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.660117097s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.627496201s trace: https://jaeger.dev.storj.io/trace/7b1dbf6a8934eee6 FAIL (2025-12-10 23:36:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.627519638s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 1m0.10699489s trace: https://jaeger.dev.storj.io/trace/67dc48038e9b3a04 FAIL (2025-12-10 23:36:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== PBD3OQ7B54A, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.520704771 (128h7m8.501057417s) [noise=true , address=77.90.31.12:4619 , piece_id=6V46S3QZAUFNOG7DXYC63IHZQN6ZOK7PQRS4HHMW7Y6WQDQSVDLA, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.521264303 (128h7m8.501616945s) [noise=true , address=145.79.227.47:4280 , piece_id=DH7CVYMKH7ZKSXAKHRBXSULW5YGMWEL46AYPCH4R5XA2K7AS5QOQ, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.521352366 (128h7m8.501705003s) [noise=true , address=45.91.116.7:4021 , piece_id=56MHLKMAQTKVHGASZB2XZY5F2TIWHTKPQDLNIV57UEYIHLCOQW6Q, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.521656722 (128h7m8.502009358s) [noise=true , address=64.49.13.26:4953 , piece_id=CH44SFUXFDC6374PDZNG3ZWJVV5GHRW6UVRCJHG7CNUNAEIH5LUQ, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.521975634 (128h7m8.502328271s) [noise=true , address=45.91.116.6:4993 , piece_id=PC4QDIQ7FW2NKWB3H2DADRX4XVIAISTHAKB72FQDNOUWF3RLQAMA, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.522004073 (128h7m8.502356719s) [noise=true , address=45.91.116.4:4944 , piece_id=T7KGZI6L43WCU3M5GMAYWNWYYIFQG4A3MNMP5JVZLXHID5ADL3IA, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.522349413 (128h7m8.502702058s) [noise=true , address=77.90.31.16:4737 , piece_id=VW2X3VTK5HM52JH4SIHLDBJ5G2SCSPHGFFS5UERVR6MHZ2G72KOQ, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.522374900 (128h7m8.502727536s) [noise=true , address=145.79.227.47:4285 , piece_id=4BFZQK254EJGSYMTRQ2A4OM7SRBLZUN6T5KFXLYF7O36WR7WDCYA, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.522664793 (128h7m8.503017432s) [noise=true , address=45.91.116.9:4271 , piece_id=2F44BOU7374JPYIPZU3WZTQJAGB7N74GJPS2QNZFERKFUZJQZYEQ, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.522717808 (128h7m8.503070448s) [noise=true , address=77.90.31.10:4524 , piece_id=5LSWGTJQQURPE7ZKTGO5W6JMIYD46F2S3Z4BEBLDGWSJWWS4FVJQ, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.522986585 (128h7m8.503339221s) [noise=true , address=77.90.31.18:4815 , piece_id=TMBH33MIATX4GPRP3PMS3SH3LZYO75RWA2JIIOA23DFI7EQKGGDA, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.523049165 (128h7m8.503401803s) [noise=true , address=77.90.31.15:4704 , piece_id=2WGWG4J7ZNL6L5VV24BUBGMQEFPYGIIXNYXGDZ7WLAAHSAXCI4NA, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.523392153 (128h7m8.503744792s) [noise=true , address=77.90.31.13:4651 , piece_id=UHRRPRRFRX24RJHVP7FCS3O5KZNWFLTENOZMMEMVTK54UD3CI4MA, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.523866107 (128h7m8.504218753s) [noise=true , address=77.90.31.14:4678 , piece_id=AECX3T4F7CMS47WK4ENIIQVBQ2FXIC3TLVSCSIZSJ7SRZZ327LVA, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.524385847 (128h7m8.504738492s) [noise=true , address=145.79.227.34:4919 , piece_id=NZB6K4JEGVICEWZJAUMS6BH2RASQ77PC2EKJMNCAWI6WGNUEJ4KQ, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.525325920 (128h7m8.505678565s) [noise=true , address=145.79.227.28:4734 , piece_id=LZ7B5SD2E7PYP6AP6AKDBBY3LY5RJ5IHVPWBJYNUYTSUD26LS2TA, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:28.525782778 (128h7m8.506135425s) [noise=true , address=77.90.31.21:4912 , piece_id=CJIP3J2QQXVK3ESKPPG7M7XOZCSIASBUY7JTICTZVRSY3BDVKZEQ, seg_pos={0 0}, upload=112619]: Uploading piece... Dec 10 23:35:29.071497156 (128h7m9.051849783s) [upload=112619]: Next returned nil segment Dec 10 23:35:29.071510801 (128h7m9.051863437s) [upload=112619]: Waiting for error group managing segments... Dec 10 23:35:29.216261317 (128h7m9.196613966s) [noise=true , address=145.79.227.47:4280 , piece_id=DH7CVYMKH7ZKSXAKHRBXSULW5YGMWEL46AYPCH4R5XA2K7AS5QOQ, seg_pos={0 0}, upload=112619]: Done uploading piece. err: <nil> Dec 10 23:35:29.226391302 (128h7m9.206743938s) [noise=true , address=45.91.116.7:4021 , piece_id=56MHLKMAQTKVHGASZB2XZY5F2TIWHTKPQDLNIV57UEYIHLCOQW6Q, seg_pos={0 0}, upload=112619]: Done uploading piece. err: <nil> Dec 10 23:35:29.227142817 (128h7m9.207495461s) [noise=true , address=45.91.116.4:4944 , piece_id=T7KGZI6L43WCU3M5GMAYWNWYYIFQG4A3MNMP5JVZLXHID5ADL3IA, seg_pos={0 0}, upload=112619]: Done uploading piece. err: <nil> Dec 10 23:35:29.235300326 (128h7m9.215652962s) [noise=true , address=45.91.116.6:4993 , piece_id=PC4QDIQ7FW2NKWB3H2DADRX4XVIAISTHAKB72FQDNOUWF3RLQAMA, seg_pos={0 0}, upload=112619]: Done uploading piece. err: <nil> Dec 10 23:35:29.239985007 (128h7m9.220337644s) [noise=true , address=145.79.227.47:4285 , piece_id=4BFZQK254EJGSYMTRQ2A4OM7SRBLZUN6T5KFXLYF7O36WR7WDCYA, seg_pos={0 0}, upload=112619]: Done uploading piece. err: <nil> Dec 10 23:35:29.258541766 (128h7m9.238894402s) [noise=true , address=145.79.227.28:4734 , piece_id=LZ7B5SD2E7PYP6AP6AKDBBY3LY5RJ5IHVPWBJYNUYTSUD26LS2TA, seg_pos={0 0}, upload=112619]: Done uploading piece. err: <nil> Dec 10 23:35:29.264881799 (128h7m9.245234435s) [noise=true , address=45.91.116.9:4271 , piece_id=2F44BOU7374JPYIPZU3WZTQJAGB7N74GJPS2QNZFERKFUZJQZYEQ, seg_pos={0 0}, upload=112619]: Done uploading piece. err: <nil> Dec 10 23:35:29.320670079 (128h7m9.301022715s) [noise=true , address=145.79.227.34:4919 , piece_id=NZB6K4JEGVICEWZJAUMS6BH2RASQ77PC2EKJMNCAWI6WGNUEJ4KQ, seg_pos={0 0}, upload=112619]: Done uploading piece. err: <nil> Dec 10 23:35:29.572624687 (128h7m9.552977328s) [noise=true , address=45.91.116.14:4203 , piece_id=V74V6TSKJJEIQOBAXBWL42RFW5B5MTU5RK2GWX55FPBD3OQ7B54A, seg_pos={0 0}, upload=112619]: Done uploading piece. err: <nil> Dec 10 23:35:29.859825996 (128h7m9.840178633s) [noise=true , address=64.49.13.26:4953 , piece_id=CH44SFUXFDC6374PDZNG3ZWJVV5GHRW6UVRCJHG7CNUNAEIH5LUQ, seg_pos={0 0}, upload=112619]: Done uploading piece. err: <nil> Dec 10 23:36:28.111839593 (128h8m8.092192232s) [noise=true , address=77.90.31.16:4737 , piece_id=VW2X3VTK5HM52JH4SIHLDBJ5G2SCSPHGFFS5UERVR6MHZ2G72KOQ, seg_pos={0 0}, upload=112619]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 139.59.144.22:49790->77.90.31.16:4737: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:28.111983685 (128h8m8.092336333s) [noise=true , address=77.90.31.13:4651 , piece_id=UHRRPRRFRX24RJHVP7FCS3O5KZNWFLTENOZMMEMVTK54UD3CI4MA, seg_pos={0 0}, upload=112619]: Done uploading piece. err: ecclient: upload failed (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o, address:77.90.31.13:4651): protocol: rpc: dial tcp 77.90.31.13:4651: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:28.112063031 (128h8m8.092415677s) [noise=true , address=77.90.31.18:4815 , piece_id=TMBH33MIATX4GPRP3PMS3SH3LZYO75RWA2JIIOA23DFI7EQKGGDA, seg_pos={0 0}, upload=112619]: Done uploading piece. err: ecclient: upload failed (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw, address:77.90.31.18:4815): protocol: rpc: dial tcp 77.90.31.18:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:28.112264972 (128h8m8.092617617s) [noise=true , address=77.90.31.15:4704 , piece_id=2WGWG4J7ZNL6L5VV24BUBGMQEFPYGIIXNYXGDZ7WLAAHSAXCI4NA, seg_pos={0 0}, upload=112619]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: rpc: dial tcp 77.90.31.15:4704: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:28.112327586 (128h8m8.092680234s) [noise=true , address=77.90.31.21:4912 , piece_id=CJIP3J2QQXVK3ESKPPG7M7XOZCSIASBUY7JTICTZVRSY3BDVKZEQ, seg_pos={0 0}, upload=112619]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: rpc: dial tcp 77.90.31.21:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:28.112456393 (128h8m8.092809055s) [noise=true , address=77.90.31.14:4678 , piece_id=AECX3T4F7CMS47WK4ENIIQVBQ2FXIC3TLVSCSIZSJ7SRZZ327LVA, seg_pos={0 0}, upload=112619]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: write tcp 139.59.144.22:44456->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:28.112547650 (128h8m8.092900298s) [noise=true , address=77.90.31.12:4619 , piece_id=6V46S3QZAUFNOG7DXYC63IHZQN6ZOK7PQRS4HHMW7Y6WQDQSVDLA, seg_pos={0 0}, upload=112619]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:60474->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:28.112762317 (128h8m8.093114964s) [noise=true , address=77.90.31.10:4524 , piece_id=5LSWGTJQQURPE7ZKTGO5W6JMIYD46F2S3Z4BEBLDGWSJWWS4FVJQ, seg_pos={0 0}, upload=112619]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 139.59.144.22:45916->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:28.112798899 (128h8m8.093151536s) [seg_pos={0 0}, upload=112619]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:36:28.217244541 (128h8m8.197597179s) [upload=112619]: Done uploading segments... Dec 10 23:36:28.217291981 (128h8m8.197644624s) [upload=112619]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.107011945s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........................................XXX..............] duration: 2.738304175s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:36:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.738343052s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................................XXX............] duration: 1.627118095s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:36:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.627148306s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.503250474s trace: https://jaeger.dev.storj.io/trace/7b1dbf6a8934eee6 FAIL (2025-12-10 23:35:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.503340512s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............................................XXX.............] duration: 2.995253544s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:35:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.995273641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 1m0.111808842s trace: https://jaeger.dev.storj.io/trace/2ae56f59e56629b FAIL (2025-12-10 23:35:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== upload segment... Dec 10 23:34:28.278761894 (128h6m8.259114531s) [seg_pos={0 0}, upload=112606]: Done begin upload segment. Dec 10 23:34:28.278801126 (128h6m8.259153773s) [noise=true , address=77.90.31.15:4700 , piece_id=4ZDOTCDRNP6J3EITTXWLZYPG7MVCNUPJUT5HJSPYZ2US5E3FB74Q, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.278823712 (128h6m8.259176356s) [noise=true , address=45.91.116.10:4090 , piece_id=5T5T64SDS5VCSVZWPYBBR2ASOFHXT4X3Y5HGJ44I2ERVAIFRLIDQ, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.279311357 (128h6m8.259663986s) [noise=true , address=77.90.31.15:4704 , piece_id=FTBQPUMUFCFRIWVEJEIN6FIVJARSXSCOLGLYEBRUPAXS7ATWKNPQ, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.279743719 (128h6m8.260096357s) [noise=true , address=145.79.227.49:4356 , piece_id=AZ4H3RBO7AOGYBSW4C2YRWDXLGXF54BJMYX4M7R7R4AYJE4DTFCQ, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.280186473 (128h6m8.260539112s) [noise=true , address=45.91.116.4:4942 , piece_id=E2DAEWFLM3BA5HZTPCDUVNIDSN33NWCBBRLN7AHKDEN3PKGFCCJA, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.280685541 (128h6m8.261038186s) [noise=true , address=77.90.31.10:4543 , piece_id=IWOPEHVFQBWIJ7TIIMK525MH7M3PTUUFLFEMP3TUABN7UHPELIQA, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.281216937 (128h6m8.261569579s) [noise=true , address=45.91.116.5:4308 , piece_id=NAFZEWTTTDI6ZR344BRATPZGSDMMI2JKSRUOQ3XKZXOV5B3ER7PA, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.281902466 (128h6m8.262255108s) [noise=true , address=64.49.13.154:4925 , piece_id=ZMVWMSTIPWZPKVWA3572VAZL4Z7UHW63MQENTTZM7JCQIGGDO3DA, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.282437531 (128h6m8.262790159s) [noise=true , address=77.90.31.13:4629 , piece_id=TZ3Z2ZBTOPFWGI7MD6UBEWDTH2QEZCOXG27BAQOSH5ZPVHXBNVOQ, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.282916030 (128h6m8.263268673s) [noise=true , address=145.79.227.2:4148 , piece_id=3BFF4ZWYA4RN5ZQJ5SMVOFNIZVVM24Z57RZZIFEJX453ASUTCDNA, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.283246585 (128h6m8.263599233s) [noise=true , address=5.231.73.5:4020 , piece_id=VVQMZD6NPW5V6WKJL6T42TOOFVF4NJN5J3JWB5QHBJISJZATKG4Q, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.283745599 (128h6m8.26409824s) [noise=true , address=77.90.31.16:4754 , piece_id=6OR2VH2KPV2ZDQKZA7BVEMRNQ6INI32M7VIJWETZOOJJ3II3EHNA, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.284220917 (128h6m8.264573562s) [noise=true , address=77.90.31.11:4582 , piece_id=E2JTLRQCNYHMKERB7AJOVREASVNGNWYYPD4I5JSXUDBP6ULQFFSA, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.284705173 (128h6m8.265057821s) [noise=true , address=45.91.116.13:4158 , piece_id=5QCLSGGRU4OYZRHNDDMOWHS6KU737UQ47HWY45QNGJCN2GFZDY5Q, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.285907311 (128h6m8.266259953s) [noise=true , address=145.79.227.30:4767 , piece_id=EURZJ2HLQUQBBV3MG5XHBSHVBXUYPRI23F74RI6BRMFVY6FTOQ2A, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.286247230 (128h6m8.266599868s) [noise=true , address=64.49.13.170:4727 , piece_id=GYWD3U6YYM6WAR2YTOXT7HJDILFKR54FYENA6KVLFSRDPBIC7AAQ, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.286642340 (128h6m8.266994981s) [noise=true , address=145.79.227.44:4228 , piece_id=IWYKODSAKYFQ36W5OKH2XWCYPPJVB4CTEFYUZ7IPXZBZMVBQGHDQ, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.287050909 (128h6m8.267403555s) [noise=true , address=77.90.31.19:4854 , piece_id=LLDI3XVW5QWPTCJVYVQCNVCBQ2XUPACYJM6O6LXQWLSQCSSGLAWA, seg_pos={0 0}, upload=112606]: Uploading piece... Dec 10 23:34:28.828061857 (128h6m8.808414483s) [upload=112606]: Next returned nil segment Dec 10 23:34:28.828079417 (128h6m8.808432061s) [upload=112606]: Waiting for error group managing segments... Dec 10 23:34:28.992190117 (128h6m8.972542754s) [noise=true , address=45.91.116.4:4942 , piece_id=E2DAEWFLM3BA5HZTPCDUVNIDSN33NWCBBRLN7AHKDEN3PKGFCCJA, seg_pos={0 0}, upload=112606]: Done uploading piece. err: <nil> Dec 10 23:34:28.992737397 (128h6m8.973090034s) [noise=true , address=145.79.227.2:4148 , piece_id=3BFF4ZWYA4RN5ZQJ5SMVOFNIZVVM24Z57RZZIFEJX453ASUTCDNA, seg_pos={0 0}, upload=112606]: Done uploading piece. err: <nil> Dec 10 23:34:28.999409844 (128h6m8.979762481s) [noise=true , address=45.91.116.10:4090 , piece_id=5T5T64SDS5VCSVZWPYBBR2ASOFHXT4X3Y5HGJ44I2ERVAIFRLIDQ, seg_pos={0 0}, upload=112606]: Done uploading piece. err: <nil> Dec 10 23:34:29.004606972 (128h6m8.984959612s) [noise=true , address=45.91.116.13:4158 , piece_id=5QCLSGGRU4OYZRHNDDMOWHS6KU737UQ47HWY45QNGJCN2GFZDY5Q, seg_pos={0 0}, upload=112606]: Done uploading piece. err: <nil> Dec 10 23:34:29.018968437 (128h6m8.999321078s) [noise=true , address=45.91.116.5:4308 , piece_id=NAFZEWTTTDI6ZR344BRATPZGSDMMI2JKSRUOQ3XKZXOV5B3ER7PA, seg_pos={0 0}, upload=112606]: Done uploading piece. err: <nil> Dec 10 23:34:29.210982273 (128h6m9.19133491s) [noise=true , address=5.231.73.5:4020 , piece_id=VVQMZD6NPW5V6WKJL6T42TOOFVF4NJN5J3JWB5QHBJISJZATKG4Q, seg_pos={0 0}, upload=112606]: Done uploading piece. err: <nil> Dec 10 23:34:29.371499198 (128h6m9.351851835s) [noise=true , address=145.79.227.49:4356 , piece_id=AZ4H3RBO7AOGYBSW4C2YRWDXLGXF54BJMYX4M7R7R4AYJE4DTFCQ, seg_pos={0 0}, upload=112606]: Done uploading piece. err: <nil> Dec 10 23:34:29.534648045 (128h6m9.515000681s) [noise=true , address=64.49.13.170:4727 , piece_id=GYWD3U6YYM6WAR2YTOXT7HJDILFKR54FYENA6KVLFSRDPBIC7AAQ, seg_pos={0 0}, upload=112606]: Done uploading piece. err: <nil> Dec 10 23:34:29.602294705 (128h6m9.582647342s) [noise=true , address=64.49.13.154:4925 , piece_id=ZMVWMSTIPWZPKVWA3572VAZL4Z7UHW63MQENTTZM7JCQIGGDO3DA, seg_pos={0 0}, upload=112606]: Done uploading piece. err: <nil> Dec 10 23:34:29.995652016 (128h6m9.97600466s) [noise=true , address=145.79.227.44:4228 , piece_id=IWYKODSAKYFQ36W5OKH2XWCYPPJVB4CTEFYUZ7IPXZBZMVBQGHDQ, seg_pos={0 0}, upload=112606]: Done uploading piece. err: <nil> Dec 10 23:34:30.001545002 (128h6m9.981897662s) [noise=true , address=145.79.227.30:4767 , piece_id=EURZJ2HLQUQBBV3MG5XHBSHVBXUYPRI23F74RI6BRMFVY6FTOQ2A, seg_pos={0 0}, upload=112606]: Done uploading piece. err: <nil> Dec 10 23:35:27.866051270 (128h7m7.846403912s) [noise=true , address=77.90.31.19:4854 , piece_id=LLDI3XVW5QWPTCJVYVQCNVCBQ2XUPACYJM6O6LXQWLSQCSSGLAWA, seg_pos={0 0}, upload=112606]: Done uploading piece. err: ecclient: upload failed (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE, address:77.90.31.19:4854): protocol: rpc: dial tcp 77.90.31.19:4854: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:35:27.866073990 (128h7m7.846426632s) [noise=true , address=77.90.31.11:4582 , piece_id=E2JTLRQCNYHMKERB7AJOVREASVNGNWYYPD4I5JSXUDBP6ULQFFSA, seg_pos={0 0}, upload=112606]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:35:27.866217752 (128h7m7.846570391s) [noise=true , address=77.90.31.13:4629 , piece_id=TZ3Z2ZBTOPFWGI7MD6UBEWDTH2QEZCOXG27BAQOSH5ZPVHXBNVOQ, seg_pos={0 0}, upload=112606]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: write tcp 139.59.144.22:34614->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:35:27.866293178 (128h7m7.846645821s) [noise=true , address=77.90.31.15:4700 , piece_id=4ZDOTCDRNP6J3EITTXWLZYPG7MVCNUPJUT5HJSPYZ2US5E3FB74Q, seg_pos={0 0}, upload=112606]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:35:27.866347214 (128h7m7.84669985s) [noise=true , address=77.90.31.16:4754 , piece_id=6OR2VH2KPV2ZDQKZA7BVEMRNQ6INI32M7VIJWETZOOJJ3II3EHNA, seg_pos={0 0}, upload=112606]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 139.59.144.22:41450->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:35:27.866475776 (128h7m7.846828414s) [noise=true , address=77.90.31.15:4704 , piece_id=FTBQPUMUFCFRIWVEJEIN6FIVJARSXSCOLGLYEBRUPAXS7ATWKNPQ, seg_pos={0 0}, upload=112606]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: write tcp 139.59.144.22:44788->77.90.31.15:4704: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:35:27.866508369 (128h7m7.84686101s) [noise=true , address=77.90.31.10:4543 , piece_id=IWOPEHVFQBWIJ7TIIMK525MH7M3PTUUFLFEMP3TUABN7UHPELIQA, seg_pos={0 0}, upload=112606]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: rpc: dial tcp 77.90.31.10:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:35:27.866533121 (128h7m7.846885757s) [seg_pos={0 0}, upload=112606]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:35:27.974801784 (128h7m7.955154421s) [upload=112606]: Done uploading segments... Dec 10 23:35:27.974835466 (128h7m7.955188105s) [upload=112606]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.111822213s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................................XXX...........] duration: 1.597568681s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:35:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.597594576s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............................................XXX............] duration: 2.680576566s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:35:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.680603344s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.768997528s trace: https://jaeger.dev.storj.io/trace/7b1dbf6a8934eee6 FAIL (2025-12-10 23:34:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.769229961s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................................XXX..........] duration: 1.569008944s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:34:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.569172321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 1m0.11193322s trace: https://jaeger.dev.storj.io/trace/79dc4b2e22c7231a FAIL (2025-12-10 23:34:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 7.905677496 (128h5m7.886030133s) [upload=112589]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:33:28.022695594 (128h5m8.00304824s) [seg_pos={0 0}, upload=112589]: Begin upload segment... Dec 10 23:33:28.022773002 (128h5m8.003125644s) [seg_pos={0 0}, upload=112589]: Done begin upload segment. Dec 10 23:33:28.022816970 (128h5m8.003169616s) [noise=true , address=145.79.227.33:4889 , piece_id=UON7WHK3JPTV7D4V5WY6K4W3P7BVGRDJ6IWIGI7BIY4V33DNFFVA, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.023933452 (128h5m8.004286102s) [noise=true , address=77.90.31.12:4616 , piece_id=PJ5YRINC32ERK6YRYOG5Y62ZVVFD4726MBJBGOD6LQGGU3H6C4GA, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.023948881 (128h5m8.00430151s) [noise=true , address=64.49.13.26:4952 , piece_id=GTLIBFGO2XUGU545K5PRA7VJVAZHBVEPDAMUTRYV5E4UPWH2AW4A, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.024446096 (128h5m8.004798746s) [noise=true , address=5.231.73.4:4007 , piece_id=IUJHOCGDSF3JM322NNLCLGQU2GXNRJYMJGPJG7EXOR4ORCTMYYVA, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.024642169 (128h5m8.004994816s) [noise=true , address=77.90.31.16:4746 , piece_id=HLF2DGOJYRAFUMJNUFOPBYMSV3Y66LFLQXF53H5HUUX5VD73XLIA, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.024941780 (128h5m8.005294423s) [noise=true , address=77.90.31.14:4678 , piece_id=WKXLGAKAR2XRUATQPC6QAAIKUB6OODHVDTRNHA7USPSW22RQVNIQ, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.026105942 (128h5m8.00645858s) [noise=true , address=145.79.227.26:4680 , piece_id=7S7KJMMYY26377XEWR5JD7KEOH3RWURAIC62WSHQ4IX4MIKC4N3Q, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.026650305 (128h5m8.007002977s) [noise=true , address=77.90.31.10:4541 , piece_id=UTG4OXP6CARSS3VGBGGT67VPGQRH5CTF7ELR22GJ63CD7FWCFFRA, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.027211252 (128h5m8.0075639s) [noise=true , address=145.79.227.50:4380 , piece_id=UPKVPMJKQNWLS4NNCTT52VCBUMZHIBP2GLCUQPHIE2KEGTAMFQTQ, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.027993935 (128h5m8.008346575s) [noise=true , address=5.231.73.5:4011 , piece_id=E7IBVIDMEFPBWNHP4RDQSN2HJ7YVCJVXDIORX4NA3GBOIY37CADA, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.029035198 (128h5m8.009387835s) [noise=true , address=45.91.116.10:4092 , piece_id=4FV7HVNPFIDYPL22LSWFVVMZI5ZVXC6KI2I2BAHPDHK7V2MPZ4WQ, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.029676205 (128h5m8.01002885s) [noise=true , address=145.79.227.30:4787 , piece_id=D7R6PP4MHLRJKZVZ77M7F25PPB3PXEXFRE6BLTZWGLB2R5PTMIAQ, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.030732101 (128h5m8.011084748s) [noise=true , address=77.90.31.11:4561 , piece_id=2WPBPS6N47BGJUMALPZOYZSKYC6IDB6R63JJ7KNNUM75TUM3EDFA, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.031116652 (128h5m8.011469297s) [noise=true , address=145.79.227.52:4470 , piece_id=XJUNM474QGGLPBJMB2NN3FLWCQHCIQHOVAJWCDSEJNWUEYSD5MPA, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.032047061 (128h5m8.012399702s) [noise=true , address=45.91.116.14:4197 , piece_id=PQ2EBILJIN6MYO2O34HUDMVEZPKYP3RQVQDXU7B5CXLPZB2EBCMA, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.032244307 (128h5m8.01259695s) [noise=true , address=77.90.31.13:4630 , piece_id=PKG3ZW26KFHW4QE6DATMEX3QV3BESQYCWVZHPCOXPZ4BVQDYE32Q, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.033466250 (128h5m8.013818897s) [noise=true , address=45.91.116.11:4126 , piece_id=X5GC4EN75RLSNRKC3YFPOL74467VSDR6D77AMD4M5HUDBFQLFMXA, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.035103141 (128h5m8.015455778s) [noise=true , address=45.91.116.9:4289 , piece_id=DFP543ESQLUHBWTHIPRGESY3ZXWF5VFWHROK64GMR2TT6KTRK2GQ, seg_pos={0 0}, upload=112589]: Uploading piece... Dec 10 23:33:28.583361263 (128h5m8.563713907s) [upload=112589]: Next returned nil segment Dec 10 23:33:28.583378986 (128h5m8.563731631s) [upload=112589]: Waiting for error group managing segments... Dec 10 23:33:28.727807548 (128h5m8.708160184s) [noise=true , address=145.79.227.52:4470 , piece_id=XJUNM474QGGLPBJMB2NN3FLWCQHCIQHOVAJWCDSEJNWUEYSD5MPA, seg_pos={0 0}, upload=112589]: Done uploading piece. err: <nil> Dec 10 23:33:28.734511283 (128h5m8.714863919s) [noise=true , address=145.79.227.33:4889 , piece_id=UON7WHK3JPTV7D4V5WY6K4W3P7BVGRDJ6IWIGI7BIY4V33DNFFVA, seg_pos={0 0}, upload=112589]: Done uploading piece. err: <nil> Dec 10 23:33:28.739029302 (128h5m8.719381941s) [noise=true , address=145.79.227.50:4380 , piece_id=UPKVPMJKQNWLS4NNCTT52VCBUMZHIBP2GLCUQPHIE2KEGTAMFQTQ, seg_pos={0 0}, upload=112589]: Done uploading piece. err: <nil> Dec 10 23:33:28.757786155 (128h5m8.738138792s) [noise=true , address=45.91.116.10:4092 , piece_id=4FV7HVNPFIDYPL22LSWFVVMZI5ZVXC6KI2I2BAHPDHK7V2MPZ4WQ, seg_pos={0 0}, upload=112589]: Done uploading piece. err: <nil> Dec 10 23:33:28.759162095 (128h5m8.739514872s) [noise=true , address=45.91.116.14:4197 , piece_id=PQ2EBILJIN6MYO2O34HUDMVEZPKYP3RQVQDXU7B5CXLPZB2EBCMA, seg_pos={0 0}, upload=112589]: Done uploading piece. err: <nil> Dec 10 23:33:28.761608174 (128h5m8.741960814s) [noise=true , address=45.91.116.11:4126 , piece_id=X5GC4EN75RLSNRKC3YFPOL74467VSDR6D77AMD4M5HUDBFQLFMXA, seg_pos={0 0}, upload=112589]: Done uploading piece. err: <nil> Dec 10 23:33:28.766778796 (128h5m8.747131444s) [noise=true , address=145.79.227.26:4680 , piece_id=7S7KJMMYY26377XEWR5JD7KEOH3RWURAIC62WSHQ4IX4MIKC4N3Q, seg_pos={0 0}, upload=112589]: Done uploading piece. err: <nil> Dec 10 23:33:28.770871322 (128h5m8.751223961s) [noise=true , address=45.91.116.9:4289 , piece_id=DFP543ESQLUHBWTHIPRGESY3ZXWF5VFWHROK64GMR2TT6KTRK2GQ, seg_pos={0 0}, upload=112589]: Done uploading piece. err: <nil> Dec 10 23:33:28.882810114 (128h5m8.863162753s) [noise=true , address=5.231.73.5:4011 , piece_id=E7IBVIDMEFPBWNHP4RDQSN2HJ7YVCJVXDIORX4NA3GBOIY37CADA, seg_pos={0 0}, upload=112589]: Done uploading piece. err: <nil> Dec 10 23:33:28.915055847 (128h5m8.895408484s) [noise=true , address=5.231.73.4:4007 , piece_id=IUJHOCGDSF3JM322NNLCLGQU2GXNRJYMJGPJG7EXOR4ORCTMYYVA, seg_pos={0 0}, upload=112589]: Done uploading piece. err: <nil> Dec 10 23:33:29.266893879 (128h5m9.247246517s) [noise=true , address=64.49.13.26:4952 , piece_id=GTLIBFGO2XUGU545K5PRA7VJVAZHBVEPDAMUTRYV5E4UPWH2AW4A, seg_pos={0 0}, upload=112589]: Done uploading piece. err: <nil> Dec 10 23:33:29.708732338 (128h5m9.689084977s) [noise=true , address=145.79.227.30:4787 , piece_id=D7R6PP4MHLRJKZVZ77M7F25PPB3PXEXFRE6BLTZWGLB2R5PTMIAQ, seg_pos={0 0}, upload=112589]: Done uploading piece. err: <nil> Dec 10 23:34:27.607680888 (128h6m7.588033525s) [noise=true , address=77.90.31.16:4746 , piece_id=HLF2DGOJYRAFUMJNUFOPBYMSV3Y66LFLQXF53H5HUUX5VD73XLIA, seg_pos={0 0}, upload=112589]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: rpc: dial tcp 77.90.31.16:4746: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:27.607754915 (128h6m7.58810756s) [noise=true , address=77.90.31.14:4678 , piece_id=WKXLGAKAR2XRUATQPC6QAAIKUB6OODHVDTRNHA7USPSW22RQVNIQ, seg_pos={0 0}, upload=112589]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:27.607885342 (128h6m7.588237979s) [noise=true , address=77.90.31.12:4616 , piece_id=PJ5YRINC32ERK6YRYOG5Y62ZVVFD4726MBJBGOD6LQGGU3H6C4GA, seg_pos={0 0}, upload=112589]: Done uploading piece. err: ecclient: upload failed (node:12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, address:77.90.31.12:4616): protocol: write tcp 139.59.144.22:54546->77.90.31.12:4616: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:27.607972640 (128h6m7.588325286s) [noise=true , address=77.90.31.11:4561 , piece_id=2WPBPS6N47BGJUMALPZOYZSKYC6IDB6R63JJ7KNNUM75TUM3EDFA, seg_pos={0 0}, upload=112589]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 139.59.144.22:45672->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:27.608193669 (128h6m7.588546308s) [noise=true , address=77.90.31.13:4630 , piece_id=PKG3ZW26KFHW4QE6DATMEX3QV3BESQYCWVZHPCOXPZ4BVQDYE32Q, seg_pos={0 0}, upload=112589]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: write tcp 139.59.144.22:60374->77.90.31.13:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:27.608288125 (128h6m7.588640753s) [noise=true , address=77.90.31.10:4541 , piece_id=UTG4OXP6CARSS3VGBGGT67VPGQRH5CTF7ELR22GJ63CD7FWCFFRA, seg_pos={0 0}, upload=112589]: Done uploading piece. err: ecclient: upload failed (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW, address:77.90.31.10:4541): protocol: write tcp 139.59.144.22:33122->77.90.31.10:4541: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:27.608333151 (128h6m7.588685777s) [seg_pos={0 0}, upload=112589]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:34:27.716200081 (128h6m7.696552728s) [upload=112589]: Done uploading segments... Dec 10 23:34:27.716271816 (128h6m7.696624463s) [upload=112589]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.111948471s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.866814645s trace: https://jaeger.dev.storj.io/trace/7b1dbf6a8934eee6 FAIL (2025-12-10 23:34:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.866842801s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............................................XXX...........] duration: 3.323797499s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:34:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.323823448s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.632871621s trace: https://jaeger.dev.storj.io/trace/7b1dbf6a8934eee6 FAIL (2025-12-10 23:33:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.632897389s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................................XXX.........] duration: 1.643076763s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:33:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.643103955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 1m0.109643426s trace: https://jaeger.dev.storj.io/trace/5241d062eec3e730 FAIL (2025-12-10 23:33:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 55 (128h4m7.755081994s) [seg_pos={0 0}, upload=112570]: Done begin upload segment. Dec 10 23:32:27.774786353 (128h4m7.755138989s) [noise=true , address=145.79.227.6:4282 , piece_id=MTRIQ7SRKY63WECD5JG7PCTHC6XHSKJNIEJTAA3NGFQQ725DW7OQ, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.774812765 (128h4m7.75516541s) [noise=true , address=64.49.13.169:4709 , piece_id=W63HM23OY5TGYYMYQ3BOAQOW3JXQNAY4P6XBPAPOSA4VJ3LRLQQQ, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.775464498 (128h4m7.755817146s) [noise=true , address=77.90.31.15:4714 , piece_id=MLNC7JEJFLQPDP52PRSKQJAJMUF5UQX6JWPZ5EYDSQCGSDQ33AWA, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.778462671 (128h4m7.758815301s) [noise=true , address=5.231.73.5:4023 , piece_id=FTILCVFEJW5SPD4KS2ZYBU2VNTXB6PBP6AYUOOQBQ3S6DGWTZ5XQ, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.779036097 (128h4m7.759388745s) [noise=true , address=145.79.227.25:4646 , piece_id=SELDJ5VIWQTMMZAWRHEOKGWCILJM2EZFZTAIZRW7PKJLL2WJZLMA, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.779397366 (128h4m7.759749995s) [noise=true , address=145.79.227.4:4188 , piece_id=OF77VDXHBBGX44UGA6ETQFEOLY7XLVDE7OS7B4NO4TGZWJYLOJGQ, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.779918030 (128h4m7.760270653s) [noise=true , address=45.91.116.14:4201 , piece_id=IWEGLCGP55V4CYUCSM3GM45Z5PUKS5YOPSQR4EOCMCGAT5JKLXBQ, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.780270930 (128h4m7.760623559s) [noise=true , address=77.90.31.13:4639 , piece_id=PVPA6GI2PROPR4D26NOQDXYUWSGMWKUHC4UOQX6GJHJ6NCPTUOUA, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.780731590 (128h4m7.761084219s) [noise=true , address=145.79.227.3:4163 , piece_id=ORWCFZFMXW6E7K2W7LLKHNGLSMPKZIJUEDVSKLUKKXL3YXCB6WRA, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.782813075 (128h4m7.763165716s) [noise=true , address=77.90.31.17:4779 , piece_id=WKVKEWXRD7GY25BRBQCPEWNNY2DR7WYKUZPY7GSNBVICIGLG4VUQ, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.784239053 (128h4m7.764591682s) [noise=true , address=77.90.31.10:4537 , piece_id=WZVSJRMVVVC45KMLBNZISQQVONZEU5WEOCNZ4D4LWMBI5YWLDPCQ, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.784832488 (128h4m7.765185118s) [noise=true , address=145.79.227.31:4815 , piece_id=F6JVMXMGXHGF6AGBPWQV4TYR6DA7U6226LIIDH65ZYOER6JWD5EQ, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.785383867 (128h4m7.765736498s) [noise=true , address=77.90.31.14:4685 , piece_id=J6H6UEZ7NAE4GSPMVGLLCPKSNOI5NJXPV4KO3VCBWVNX4XQ62Z4A, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.785617907 (128h4m7.765970538s) [noise=true , address=45.91.116.10:4107 , piece_id=AK32HDWDPBNKSXUBUYIK2MZKZJLECC5E2CKLYE4B4WLBOCB6QHQQ, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.786429828 (128h4m7.766782465s) [noise=true , address=145.79.227.44:4215 , piece_id=QJ2XHXPBYVGIRJPEQGF5PVKIZLTAEHZ3XGQFAHZKBOSG66L4LLPA, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.786931163 (128h4m7.767283791s) [noise=true , address=77.90.31.21:4902 , piece_id=TWOM5WGBSQLA2OHWZMMJBGZS7JJV22QQDMXSH5MNDVNA73J3EKOA, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.787474507 (128h4m7.767827136s) [noise=true , address=45.91.116.12:4241 , piece_id=UNBZ7V4TOVTV3LQE3NZFVDUXG2SBT5H3ODFACUYWJJ6AAHG2ODCQ, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:27.788048289 (128h4m7.768400912s) [noise=true , address=77.90.31.18:4834 , piece_id=AAP4RNFYOJJDKTSSHQ4INXGOSZQSB5NQX7VQJCEG2XXQKLVPPGAA, seg_pos={0 0}, upload=112570]: Uploading piece... Dec 10 23:32:28.341075608 (128h4m8.321428236s) [upload=112570]: Next returned nil segment Dec 10 23:32:28.342028667 (128h4m8.322381314s) [upload=112570]: Waiting for error group managing segments... Dec 10 23:32:28.482425454 (128h4m8.462778098s) [noise=true , address=145.79.227.6:4282 , piece_id=MTRIQ7SRKY63WECD5JG7PCTHC6XHSKJNIEJTAA3NGFQQ725DW7OQ, seg_pos={0 0}, upload=112570]: Done uploading piece. err: <nil> Dec 10 23:32:28.502578612 (128h4m8.48293126s) [noise=true , address=145.79.227.31:4815 , piece_id=F6JVMXMGXHGF6AGBPWQV4TYR6DA7U6226LIIDH65ZYOER6JWD5EQ, seg_pos={0 0}, upload=112570]: Done uploading piece. err: <nil> Dec 10 23:32:28.503262433 (128h4m8.483615079s) [noise=true , address=145.79.227.4:4188 , piece_id=OF77VDXHBBGX44UGA6ETQFEOLY7XLVDE7OS7B4NO4TGZWJYLOJGQ, seg_pos={0 0}, upload=112570]: Done uploading piece. err: <nil> Dec 10 23:32:28.506819708 (128h4m8.487172348s) [noise=true , address=145.79.227.25:4646 , piece_id=SELDJ5VIWQTMMZAWRHEOKGWCILJM2EZFZTAIZRW7PKJLL2WJZLMA, seg_pos={0 0}, upload=112570]: Done uploading piece. err: <nil> Dec 10 23:32:28.507577662 (128h4m8.487930303s) [noise=true , address=45.91.116.12:4241 , piece_id=UNBZ7V4TOVTV3LQE3NZFVDUXG2SBT5H3ODFACUYWJJ6AAHG2ODCQ, seg_pos={0 0}, upload=112570]: Done uploading piece. err: <nil> Dec 10 23:32:28.530812069 (128h4m8.511164713s) [noise=true , address=45.91.116.14:4201 , piece_id=IWEGLCGP55V4CYUCSM3GM45Z5PUKS5YOPSQR4EOCMCGAT5JKLXBQ, seg_pos={0 0}, upload=112570]: Done uploading piece. err: <nil> Dec 10 23:32:28.535620278 (128h4m8.515972928s) [noise=true , address=45.91.116.10:4107 , piece_id=AK32HDWDPBNKSXUBUYIK2MZKZJLECC5E2CKLYE4B4WLBOCB6QHQQ, seg_pos={0 0}, upload=112570]: Done uploading piece. err: <nil> Dec 10 23:32:28.648441146 (128h4m8.628793797s) [noise=true , address=145.79.227.3:4163 , piece_id=ORWCFZFMXW6E7K2W7LLKHNGLSMPKZIJUEDVSKLUKKXL3YXCB6WRA, seg_pos={0 0}, upload=112570]: Done uploading piece. err: <nil> Dec 10 23:32:28.707565416 (128h4m8.68791806s) [noise=true , address=5.231.73.5:4023 , piece_id=FTILCVFEJW5SPD4KS2ZYBU2VNTXB6PBP6AYUOOQBQ3S6DGWTZ5XQ, seg_pos={0 0}, upload=112570]: Done uploading piece. err: <nil> Dec 10 23:32:29.474417365 (128h4m9.454770013s) [noise=true , address=145.79.227.44:4215 , piece_id=QJ2XHXPBYVGIRJPEQGF5PVKIZLTAEHZ3XGQFAHZKBOSG66L4LLPA, seg_pos={0 0}, upload=112570]: Done uploading piece. err: <nil> Dec 10 23:32:39.436966955 (128h4m19.417319594s) [noise=true , address=64.49.13.169:4709 , piece_id=W63HM23OY5TGYYMYQ3BOAQOW3JXQNAY4P6XBPAPOSA4VJ3LRLQQQ, seg_pos={0 0}, upload=112570]: Done uploading piece. err: <nil> Dec 10 23:33:27.364421613 (128h5m7.344774253s) [noise=true , address=77.90.31.18:4834 , piece_id=AAP4RNFYOJJDKTSSHQ4INXGOSZQSB5NQX7VQJCEG2XXQKLVPPGAA, seg_pos={0 0}, upload=112570]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 139.59.144.22:60400->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:27.364464723 (128h5m7.344817371s) [noise=true , address=77.90.31.15:4714 , piece_id=MLNC7JEJFLQPDP52PRSKQJAJMUF5UQX6JWPZ5EYDSQCGSDQ33AWA, seg_pos={0 0}, upload=112570]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: write tcp 139.59.144.22:59034->77.90.31.15:4714: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:27.364684231 (128h5m7.345036876s) [noise=true , address=77.90.31.17:4779 , piece_id=WKVKEWXRD7GY25BRBQCPEWNNY2DR7WYKUZPY7GSNBVICIGLG4VUQ, seg_pos={0 0}, upload=112570]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: write tcp 139.59.144.22:33442->77.90.31.17:4779: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:27.364752624 (128h5m7.34510527s) [noise=true , address=77.90.31.10:4537 , piece_id=WZVSJRMVVVC45KMLBNZISQQVONZEU5WEOCNZ4D4LWMBI5YWLDPCQ, seg_pos={0 0}, upload=112570]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: rpc: dial tcp 77.90.31.10:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:27.365049243 (128h5m7.345401886s) [noise=true , address=77.90.31.14:4685 , piece_id=J6H6UEZ7NAE4GSPMVGLLCPKSNOI5NJXPV4KO3VCBWVNX4XQ62Z4A, seg_pos={0 0}, upload=112570]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:27.365438385 (128h5m7.345791033s) [noise=true , address=77.90.31.21:4902 , piece_id=TWOM5WGBSQLA2OHWZMMJBGZS7JJV22QQDMXSH5MNDVNA73J3EKOA, seg_pos={0 0}, upload=112570]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 139.59.144.22:41436->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:27.365738552 (128h5m7.346091201s) [noise=true , address=77.90.31.13:4639 , piece_id=PVPA6GI2PROPR4D26NOQDXYUWSGMWKUHC4UOQX6GJHJ6NCPTUOUA, seg_pos={0 0}, upload=112570]: Done uploading piece. err: ecclient: upload failed (node:12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, address:77.90.31.13:4639): protocol: rpc: dial tcp 77.90.31.13:4639: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:27.365775277 (128h5m7.346127915s) [seg_pos={0 0}, upload=112570]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:33:27.471942622 (128h5m7.452295261s) [upload=112570]: Done uploading segments... Dec 10 23:33:27.471979814 (128h5m7.452332456s) [upload=112570]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.109657668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............................................XXX..........] duration: 2.843527679s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:33:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.843552378s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.006550113s trace: https://jaeger.dev.storj.io/trace/7b1dbf6a8934eee6 FAIL (2025-12-10 23:33:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.006636s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................................................XXX........] duration: 1.678538807s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:33:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.678568512s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................................................XXX.........] duration: 2.91161865s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:32:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.911660583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 1m0.139319511s trace: https://jaeger.dev.storj.io/trace/722a84b87edd4057 FAIL (2025-12-10 23:32:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2294536 (128h3m7.482647172s) [seg_pos={0 0}, upload=112556]: Done begin upload segment. Dec 10 23:31:27.502357018 (128h3m7.482709659s) [noise=true , address=77.90.31.15:4725 , piece_id=363QUC3OEOF7MQGQYJZWGYPTJ33RYT7Z7WC4Q3ADAXF3X6MH2ZZQ, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.502406630 (128h3m7.482759275s) [noise=true , address=45.91.116.10:4097 , piece_id=7TPPC7CHMOZF2QPANU37NUMFADXOZGLGJHY4Q7EW5GIZNGTE5PRQ, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.502486919 (128h3m7.482839559s) [noise=true , address=45.91.116.13:4158 , piece_id=76JVO636VRDVSZI3LUXQRHM24YNBXWVD3EWFIL2NKO46MHKANSZQ, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.503506051 (128h3m7.483858698s) [noise=true , address=77.90.31.13:4647 , piece_id=HM5DGYHXC7MSRC6ODJGNS2LLP5NVIOIHBLZFIGWK64O43RKLCY3A, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.504485130 (128h3m7.484837757s) [noise=true , address=5.231.73.3:4029 , piece_id=L7QPWSWEHZSGBFJC7CWLURUNXQN47IIMYV7AGQZA3KAC7KTKN3JQ, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.504832562 (128h3m7.485185231s) [noise=true , address=5.231.73.2:4030 , piece_id=6746QBZ453VCL6UFFWX6OF7XMMHN6WZL4KXLDYKYA4NCALUPYWWQ, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.504965388 (128h3m7.485318028s) [noise=true , address=145.79.227.30:4773 , piece_id=IR4CWSW6CETOHMGCZ22HVH2ZFFDM2YFLS2JCEMVPRNDO22NF54LA, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.505413192 (128h3m7.485765828s) [noise=true , address=145.79.227.22:4516 , piece_id=X3W6ACKBJOUZBVZTG6U6DE2WHYZWYS4G2RCMKSQPXM25KTS3YWHA, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.505566044 (128h3m7.485918683s) [noise=true , address=77.90.31.14:4676 , piece_id=7TCBBGFJKREZBMGRON5WJEOIRTVX6N5WOUWK2WYLDYVX6P4YJN5Q, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.505883701 (128h3m7.486236344s) [noise=true , address=145.79.227.39:4030 , piece_id=BG36ETIX2EIGDXVZTS3QC5QTKJTQLREWTOTJHA7IAH77F64Q5ALA, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.506144173 (128h3m7.486496812s) [noise=true , address=145.79.227.43:4163 , piece_id=SH5GXZ5PQFUCGJ5FHGM2FKOBJAYBHIWN3SK5FCBWWHUAE5XPT7VQ, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.506402821 (128h3m7.486755448s) [noise=true , address=77.90.31.16:4764 , piece_id=V5SQFQ7VZUKZLA2TNJW4ZYL4MTCVFOB6QLLNDAT5MVRVZUNIXXPQ, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.506529005 (128h3m7.486881633s) [noise=true , address=5.231.73.4:4013 , piece_id=SB4RTKQKLMOQMUCWUDRZ3SWO7TXSROOAHSFHZ6NXWCEHXJBFRB7Q, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.506870227 (128h3m7.487222872s) [noise=true , address=145.79.227.34:4919 , piece_id=EFWLNYD6AILZK6CEQW7P7PJCM3V7OPDUWVIDKGGVFFMAWOIVNIPA, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.506875944 (128h3m7.487228574s) [noise=true , address=77.90.31.10:4551 , piece_id=G4S7SI6XYN3O5GX7ZGJR3CRFKMHIFATYKQZXMYLKKXEWUM5DSW4Q, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.507273343 (128h3m7.487625979s) [noise=true , address=45.91.116.9:4277 , piece_id=5N4N4MBACWTA577LDI6VUWGL5TR5CHC2XUCMLWTGP4V3ABIAOGZA, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.507742401 (128h3m7.488095026s) [noise=true , address=77.90.31.19:4844 , piece_id=KDMKKW2LWAAKOHEJUNEDLNIRM3U537JNH3KG5FCA255JY5JQ5ZSQ, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:27.509026131 (128h3m7.489379006s) [noise=true , address=77.90.31.12:4604 , piece_id=UOYUK33O6VJUTLFA4AS5PAD4TEHXBV7C2MEQBVE3K4DJ2PXOBLSA, seg_pos={0 0}, upload=112556]: Uploading piece... Dec 10 23:31:28.066437729 (128h3m8.046790377s) [upload=112556]: Next returned nil segment Dec 10 23:31:28.066488991 (128h3m8.046841638s) [upload=112556]: Waiting for error group managing segments... Dec 10 23:31:28.203702725 (128h3m8.184055363s) [noise=true , address=45.91.116.10:4097 , piece_id=7TPPC7CHMOZF2QPANU37NUMFADXOZGLGJHY4Q7EW5GIZNGTE5PRQ, seg_pos={0 0}, upload=112556]: Done uploading piece. err: <nil> Dec 10 23:31:28.205683595 (128h3m8.186036238s) [noise=true , address=145.79.227.43:4163 , piece_id=SH5GXZ5PQFUCGJ5FHGM2FKOBJAYBHIWN3SK5FCBWWHUAE5XPT7VQ, seg_pos={0 0}, upload=112556]: Done uploading piece. err: <nil> Dec 10 23:31:28.206838158 (128h3m8.187190799s) [noise=true , address=145.79.227.39:4030 , piece_id=BG36ETIX2EIGDXVZTS3QC5QTKJTQLREWTOTJHA7IAH77F64Q5ALA, seg_pos={0 0}, upload=112556]: Done uploading piece. err: <nil> Dec 10 23:31:28.214054899 (128h3m8.194407539s) [noise=true , address=45.91.116.9:4277 , piece_id=5N4N4MBACWTA577LDI6VUWGL5TR5CHC2XUCMLWTGP4V3ABIAOGZA, seg_pos={0 0}, upload=112556]: Done uploading piece. err: <nil> Dec 10 23:31:28.221095398 (128h3m8.201448044s) [noise=true , address=145.79.227.22:4516 , piece_id=X3W6ACKBJOUZBVZTG6U6DE2WHYZWYS4G2RCMKSQPXM25KTS3YWHA, seg_pos={0 0}, upload=112556]: Done uploading piece. err: <nil> Dec 10 23:31:28.224366792 (128h3m8.204719429s) [noise=true , address=45.91.116.13:4158 , piece_id=76JVO636VRDVSZI3LUXQRHM24YNBXWVD3EWFIL2NKO46MHKANSZQ, seg_pos={0 0}, upload=112556]: Done uploading piece. err: <nil> Dec 10 23:31:28.259076245 (128h3m8.239428883s) [noise=true , address=145.79.227.34:4919 , piece_id=EFWLNYD6AILZK6CEQW7P7PJCM3V7OPDUWVIDKGGVFFMAWOIVNIPA, seg_pos={0 0}, upload=112556]: Done uploading piece. err: <nil> Dec 10 23:31:28.427957590 (128h3m8.408310229s) [noise=true , address=5.231.73.2:4030 , piece_id=6746QBZ453VCL6UFFWX6OF7XMMHN6WZL4KXLDYKYA4NCALUPYWWQ, seg_pos={0 0}, upload=112556]: Done uploading piece. err: <nil> Dec 10 23:31:28.436624079 (128h3m8.416976718s) [noise=true , address=5.231.73.3:4029 , piece_id=L7QPWSWEHZSGBFJC7CWLURUNXQN47IIMYV7AGQZA3KAC7KTKN3JQ, seg_pos={0 0}, upload=112556]: Done uploading piece. err: <nil> Dec 10 23:31:28.644373034 (128h3m8.62472566s) [noise=true , address=5.231.73.4:4013 , piece_id=SB4RTKQKLMOQMUCWUDRZ3SWO7TXSROOAHSFHZ6NXWCEHXJBFRB7Q, seg_pos={0 0}, upload=112556]: Done uploading piece. err: <nil> Dec 10 23:31:29.151177475 (128h3m9.131530118s) [noise=true , address=145.79.227.30:4773 , piece_id=IR4CWSW6CETOHMGCZ22HVH2ZFFDM2YFLS2JCEMVPRNDO22NF54LA, seg_pos={0 0}, upload=112556]: Done uploading piece. err: <nil> Dec 10 23:32:27.102158997 (128h4m7.082511643s) [noise=true , address=77.90.31.10:4551 , piece_id=G4S7SI6XYN3O5GX7ZGJR3CRFKMHIFATYKQZXMYLKKXEWUM5DSW4Q, seg_pos={0 0}, upload=112556]: Done uploading piece. err: ecclient: upload failed (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr, address:77.90.31.10:4551): protocol: rpc: dial tcp 77.90.31.10:4551: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:32:27.102379746 (128h4m7.082732388s) [noise=true , address=77.90.31.19:4844 , piece_id=KDMKKW2LWAAKOHEJUNEDLNIRM3U537JNH3KG5FCA255JY5JQ5ZSQ, seg_pos={0 0}, upload=112556]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 139.59.144.22:44630->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:32:27.102568594 (128h4m7.082921243s) [noise=true , address=77.90.31.12:4604 , piece_id=UOYUK33O6VJUTLFA4AS5PAD4TEHXBV7C2MEQBVE3K4DJ2PXOBLSA, seg_pos={0 0}, upload=112556]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 139.59.144.22:52022->77.90.31.12:4604: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:32:27.102905426 (128h4m7.083258076s) [noise=true , address=77.90.31.13:4647 , piece_id=HM5DGYHXC7MSRC6ODJGNS2LLP5NVIOIHBLZFIGWK64O43RKLCY3A, seg_pos={0 0}, upload=112556]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: rpc: dial tcp 77.90.31.13:4647: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:32:27.103322210 (128h4m7.083674931s) [noise=true , address=77.90.31.14:4676 , piece_id=7TCBBGFJKREZBMGRON5WJEOIRTVX6N5WOUWK2WYLDYVX6P4YJN5Q, seg_pos={0 0}, upload=112556]: Done uploading piece. err: ecclient: upload failed (node:1tpRh8Xvyxe72LDgR9rkUFx9qV88axb9DUYDhRnrzST2cQuHqC, address:77.90.31.14:4676): protocol: rpc: dial tcp 77.90.31.14:4676: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:32:27.103622632 (128h4m7.083975272s) [noise=true , address=77.90.31.16:4764 , piece_id=V5SQFQ7VZUKZLA2TNJW4ZYL4MTCVFOB6QLLNDAT5MVRVZUNIXXPQ, seg_pos={0 0}, upload=112556]: Done uploading piece. err: ecclient: upload failed (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, address:77.90.31.16:4764): protocol: write tcp 139.59.144.22:59170->77.90.31.16:4764: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:32:27.103802780 (128h4m7.08415542s) [noise=true , address=77.90.31.15:4725 , piece_id=363QUC3OEOF7MQGQYJZWGYPTJ33RYT7Z7WC4Q3ADAXF3X6MH2ZZQ, seg_pos={0 0}, upload=112556]: Done uploading piece. err: ecclient: upload failed (node:1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk, address:77.90.31.15:4725): protocol: write tcp 139.59.144.22:36768->77.90.31.15:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:32:27.103829763 (128h4m7.08418239s) [seg_pos={0 0}, upload=112556]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:32:27.220800259 (128h4m7.201152907s) [upload=112556]: Done uploading segments... Dec 10 23:32:27.220843912 (128h4m7.201196563s) [upload=112556]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.139335352s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................................................XXX.......] duration: 1.616256987s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:32:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.616284864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................................................XXX........] duration: 2.808622048s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:31:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.808652479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 1m0.110011689s trace: https://jaeger.dev.storj.io/trace/7b867ca0cc418168 FAIL (2025-12-10 23:31:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ec 10 23:30:27.134191708 (128h2m7.114544351s) [upload=112542]: Uploading segments... Dec 10 23:30:27.134691618 (128h2m7.115044259s) [upload=112542]: Got next segment. Inline: false Dec 10 23:30:27.134816641 (128h2m7.115169277s) [upload=112542]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:30:27.134825318 (128h2m7.115177966s) [upload=112542]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:30:27.247992836 (128h2m7.228345473s) [seg_pos={0 0}, upload=112542]: Begin upload segment... Dec 10 23:30:27.248045453 (128h2m7.228398089s) [seg_pos={0 0}, upload=112542]: Done begin upload segment. Dec 10 23:30:27.248081657 (128h2m7.228434292s) [noise=true , address=77.90.31.18:4810 , piece_id=SD7W423YWMB6AJBAXM3POPO2PBTR5UZDL3O2MYALT6JY3WGYVP5A, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.248213962 (128h2m7.228566644s) [noise=true , address=77.90.31.10:4521 , piece_id=3CRKL4FYKYFTRTYRC4WRVU33LJ7LAURSGZQIGGUBIBFRKS54MXVA, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.249264574 (128h2m7.229617228s) [noise=true , address=45.91.116.10:4110 , piece_id=JASH4PCGLOBMMHTAGJOVZ5T5UROXUR72QCPMUZMTUMREAR55Z7TQ, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.249634381 (128h2m7.229987028s) [noise=true , address=77.90.31.11:4582 , piece_id=C3TYZFXH7INSCFVFBVXU6M2QOL6FJ3CVLNZFQSUVI4RB4U2VFLSQ, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.250771901 (128h2m7.231124542s) [noise=true , address=77.90.31.12:4600 , piece_id=ZBO5NKWF5CPJERVNJR2EX7WI2AWBS5WOLBZZIER4XP2SX56AADYQ, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.250862570 (128h2m7.231215211s) [noise=true , address=145.79.227.44:4219 , piece_id=FTEC5QFMK222NHDR7ELSRMKJZHC7U6WALOICLCJOYIT2AHRTJ3JQ, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.252207964 (128h2m7.23256061s) [noise=true , address=45.91.116.12:4243 , piece_id=SJT644IPV47M72VOAUU2JXREN5TSLZ3NTOMUXYEVAL3AXAZFEHHA, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.252393811 (128h2m7.232746458s) [noise=true , address=45.91.116.14:4202 , piece_id=FI2HGY5ET7X72Z3HOVWUWJM7DZIGGZQ7PN7PJ7BVKWHINLQRNOEA, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.253718745 (128h2m7.234071389s) [noise=true , address=77.90.31.16:4751 , piece_id=FD2CZAUEC3FVBIE5TMNVHXSNJEAFHJ4ZILEI64LC472GUP3WPERQ, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.253887027 (128h2m7.234239672s) [noise=true , address=64.49.13.169:4706 , piece_id=WASMOI7VD6J7W7DEX5VYNN7BP5ASHMGYD7ZRD55FI2G75H732D4A, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.254723958 (128h2m7.235076606s) [noise=true , address=45.91.116.9:4294 , piece_id=DB4TUO3BQGQUORNCWEV3E4V2EZDF5O3MAYE3XVRSWEROSCMYER7Q, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.255270503 (128h2m7.235623149s) [noise=true , address=5.231.73.2:4036 , piece_id=MWU3MF33AVGYXMUJSQH7VOJTSW6THEF5O4M2KE4ZOV3JFP3U6RHA, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.255849641 (128h2m7.236202291s) [noise=true , address=145.79.227.26:4679 , piece_id=BHZKI656JNKYDNGCYIW3BEDA4OGLLPYVYKNENJKCBVDDQQ6Q472A, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.258605442 (128h2m7.238958089s) [noise=true , address=45.91.116.13:4158 , piece_id=RYSA5DYRY7WQHMY6AYHJ7JNPAF3YPQBK735GXX3A2T5N5EMZGEOA, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.259351303 (128h2m7.239703952s) [noise=true , address=77.90.31.13:4654 , piece_id=2X3NWUU4JXQYA67LOJ3E6D4SCIUTUGG2EVZENCNCYTSOBEMPWOAA, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.261426893 (128h2m7.241779516s) [noise=true , address=77.90.31.17:4793 , piece_id=CVSGYSS7SOB7BZWZRLC43GFMZ3EJM5K5NIGT3MBFT37I6ZH4I3PQ, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.262664351 (128h2m7.243016978s) [noise=true , address=145.79.227.43:4163 , piece_id=JV6VIRYV2ELWT2WMEWOMSJABJMIP7ZWGFXTEEH7RA4GF6EP77RNQ, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.265382706 (128h2m7.245735346s) [noise=true , address=45.91.116.4:4949 , piece_id=AAHT4VN6AMOWR4XLYB55T43UFNUXHVCLQ4SKTZM7XCQDKJZP4ZRA, seg_pos={0 0}, upload=112542]: Uploading piece... Dec 10 23:30:27.810904137 (128h2m7.791256776s) [upload=112542]: Next returned nil segment Dec 10 23:30:27.810918817 (128h2m7.791271454s) [upload=112542]: Waiting for error group managing segments... Dec 10 23:30:27.957573966 (128h2m7.937926612s) [noise=true , address=45.91.116.4:4949 , piece_id=AAHT4VN6AMOWR4XLYB55T43UFNUXHVCLQ4SKTZM7XCQDKJZP4ZRA, seg_pos={0 0}, upload=112542]: Done uploading piece. err: <nil> Dec 10 23:30:27.958247968 (128h2m7.938600608s) [noise=true , address=145.79.227.43:4163 , piece_id=JV6VIRYV2ELWT2WMEWOMSJABJMIP7ZWGFXTEEH7RA4GF6EP77RNQ, seg_pos={0 0}, upload=112542]: Done uploading piece. err: <nil> Dec 10 23:30:27.966221878 (128h2m7.946574518s) [noise=true , address=145.79.227.26:4679 , piece_id=BHZKI656JNKYDNGCYIW3BEDA4OGLLPYVYKNENJKCBVDDQQ6Q472A, seg_pos={0 0}, upload=112542]: Done uploading piece. err: <nil> Dec 10 23:30:27.967220038 (128h2m7.94757268s) [noise=true , address=45.91.116.12:4243 , piece_id=SJT644IPV47M72VOAUU2JXREN5TSLZ3NTOMUXYEVAL3AXAZFEHHA, seg_pos={0 0}, upload=112542]: Done uploading piece. err: <nil> Dec 10 23:30:27.976359910 (128h2m7.956712547s) [noise=true , address=45.91.116.14:4202 , piece_id=FI2HGY5ET7X72Z3HOVWUWJM7DZIGGZQ7PN7PJ7BVKWHINLQRNOEA, seg_pos={0 0}, upload=112542]: Done uploading piece. err: <nil> Dec 10 23:30:27.976935205 (128h2m7.957287841s) [noise=true , address=45.91.116.13:4158 , piece_id=RYSA5DYRY7WQHMY6AYHJ7JNPAF3YPQBK735GXX3A2T5N5EMZGEOA, seg_pos={0 0}, upload=112542]: Done uploading piece. err: <nil> Dec 10 23:30:27.986349568 (128h2m7.966702204s) [noise=true , address=45.91.116.9:4294 , piece_id=DB4TUO3BQGQUORNCWEV3E4V2EZDF5O3MAYE3XVRSWEROSCMYER7Q, seg_pos={0 0}, upload=112542]: Done uploading piece. err: <nil> Dec 10 23:30:27.986931971 (128h2m7.967284614s) [noise=true , address=45.91.116.10:4110 , piece_id=JASH4PCGLOBMMHTAGJOVZ5T5UROXUR72QCPMUZMTUMREAR55Z7TQ, seg_pos={0 0}, upload=112542]: Done uploading piece. err: <nil> Dec 10 23:30:28.171042223 (128h2m8.151394868s) [noise=true , address=5.231.73.2:4036 , piece_id=MWU3MF33AVGYXMUJSQH7VOJTSW6THEF5O4M2KE4ZOV3JFP3U6RHA, seg_pos={0 0}, upload=112542]: Done uploading piece. err: <nil> Dec 10 23:30:28.992467993 (128h2m8.97282062s) [noise=true , address=145.79.227.44:4219 , piece_id=FTEC5QFMK222NHDR7ELSRMKJZHC7U6WALOICLCJOYIT2AHRTJ3JQ, seg_pos={0 0}, upload=112542]: Done uploading piece. err: <nil> Dec 10 23:30:41.105392649 (128h2m21.085745293s) [noise=true , address=64.49.13.169:4706 , piece_id=WASMOI7VD6J7W7DEX5VYNN7BP5ASHMGYD7ZRD55FI2G75H732D4A, seg_pos={0 0}, upload=112542]: Done uploading piece. err: <nil> Dec 10 23:31:26.840121335 (128h3m6.820473977s) [noise=true , address=77.90.31.12:4600 , piece_id=ZBO5NKWF5CPJERVNJR2EX7WI2AWBS5WOLBZZIER4XP2SX56AADYQ, seg_pos={0 0}, upload=112542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:31:26.840432923 (128h3m6.820785568s) [noise=true , address=77.90.31.16:4751 , piece_id=FD2CZAUEC3FVBIE5TMNVHXSNJEAFHJ4ZILEI64LC472GUP3WPERQ, seg_pos={0 0}, upload=112542]: Done uploading piece. err: ecclient: upload failed (node:12Sj7nPQiL1EmwvnVCpRv1HEeDYFnknTXLNZTvhAUmHDcP2Wpuf, address:77.90.31.16:4751): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:31:26.840496920 (128h3m6.820849565s) [noise=true , address=77.90.31.17:4793 , piece_id=CVSGYSS7SOB7BZWZRLC43GFMZ3EJM5K5NIGT3MBFT37I6ZH4I3PQ, seg_pos={0 0}, upload=112542]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:31:26.840797193 (128h3m6.821149842s) [noise=true , address=77.90.31.10:4521 , piece_id=3CRKL4FYKYFTRTYRC4WRVU33LJ7LAURSGZQIGGUBIBFRKS54MXVA, seg_pos={0 0}, upload=112542]: Done uploading piece. err: ecclient: upload failed (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa, address:77.90.31.10:4521): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:31:26.841443644 (128h3m6.821796289s) [noise=true , address=77.90.31.11:4582 , piece_id=C3TYZFXH7INSCFVFBVXU6M2QOL6FJ3CVLNZFQSUVI4RB4U2VFLSQ, seg_pos={0 0}, upload=112542]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:31:26.841626431 (128h3m6.821979074s) [noise=true , address=77.90.31.18:4810 , piece_id=SD7W423YWMB6AJBAXM3POPO2PBTR5UZDL3O2MYALT6JY3WGYVP5A, seg_pos={0 0}, upload=112542]: Done uploading piece. err: ecclient: upload failed (node:12mfG3ZeKyFWEPsTgSoR4wqiUYY7YobwmMtsg1GsCorquCZLX5F, address:77.90.31.18:4810): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:31:26.841645878 (128h3m6.821998501s) [noise=true , address=77.90.31.13:4654 , piece_id=2X3NWUU4JXQYA67LOJ3E6D4SCIUTUGG2EVZENCNCYTSOBEMPWOAA, seg_pos={0 0}, upload=112542]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:31:26.841676345 (128h3m6.822028967s) [seg_pos={0 0}, upload=112542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:31:26.948154240 (128h3m6.928506863s) [upload=112542]: Done uploading segments... Dec 10 23:31:26.948206598 (128h3m6.928559237s) [upload=112542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.110046568s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................................................XXX......] duration: 1.674763006s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:31:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.674791782s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................................................XXX.......] duration: 2.83316472s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:30:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.833214716s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................................................XXX.....] duration: 2.176434532s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:30:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.176466588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 1m0.119783609s trace: https://jaeger.dev.storj.io/trace/4b29c46e800b9fe7 FAIL (2025-12-10 23:30:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== , upload=112529]: Done begin upload segment. Dec 10 23:29:24.733501410 (128h1m4.713854047s) [noise=true , address=64.49.13.169:4715 , piece_id=QD4ENFOYMQJD2BNH47M5YX2SNYCD72FENVZUJKSTKXFLOBMMVHPQ, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.734052850 (128h1m4.714405477s) [noise=true , address=145.79.227.25:4646 , piece_id=EHCBJMRGQONG66IBBLHQGMMHUSVBRDPGS3GSYFAETO3EXZRSRLGA, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.734809784 (128h1m4.715162428s) [noise=true , address=145.79.227.48:4329 , piece_id=PWR6FMYUQ6M3NR34DOZ2KXJ2BR7Z7Z2NL5GYCDMC7MM4JSQ4ISVQ, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.735321237 (128h1m4.715673883s) [noise=true , address=45.91.116.4:4959 , piece_id=BMN7VCKIYSNRFCBXUF7H6R2LBLD4IT3SQMQX4RDGNXOOURYGPBPA, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.735842791 (128h1m4.716195501s) [noise=true , address=45.91.116.5:4312 , piece_id=6ETOURPL3VSJ2HKQN4DQEIZG35YPAMD7FPITAA6X4NYJG5LV2PWQ, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.736221239 (128h1m4.716573943s) [noise=true , address=77.90.31.13:4630 , piece_id=OKZUCQYP5TXZTUTEJQ5Y3L3NMRAP5W2TXWZ5TN4DXKMF5HLFH6FA, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.736418370 (128h1m4.716771007s) [noise=true , address=77.90.31.16:4765 , piece_id=NRSFGLZJDRW7UNYNY6TGRVMB3XFHJ2N3OUC7FF2KR6Y4SNTB35JQ, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.736824556 (128h1m4.717177202s) [noise=true , address=77.90.31.12:4600 , piece_id=Y6K6SXYFBF66TLPTG3FKOU57DPMKU3KBKATSLWCLWMN7YYDU4H5Q, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.737389484 (128h1m4.71774213s) [noise=true , address=145.79.227.44:4201 , piece_id=M4NOMVMGD4H7SYKK4TFOPBEFWXBFQ2R527FGC5ZCF2DN4BS7QQAA, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.737430177 (128h1m4.717782818s) [noise=true , address=77.90.31.21:4906 , piece_id=HKZLIRO4CUMPOHO3G75JW3IXWVQ2VZXDUOC3YI7L4FXZ3PGJDLDA, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.737814537 (128h1m4.718167178s) [noise=true , address=145.79.227.29:4957 , piece_id=2YBLZFDAPOTVZRTDN662JTXKVPB44YITDJZN4T5SKEZ4HHACJKTQ, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.737968912 (128h1m4.718321571s) [noise=true , address=145.79.227.28:4757 , piece_id=AEZSB5ELGJD2R7EC37PEIYTUXFX6RZH6TRNR46OKMVNFU5P4IVZA, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.738253114 (128h1m4.718605759s) [noise=true , address=145.79.227.30:4778 , piece_id=MEC3V2IROPODBRVZWI6RVTPNZOYHXZIOKTXRHULWU56XBI6YYJ7Q, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.738687067 (128h1m4.71903971s) [noise=true , address=45.91.116.10:4091 , piece_id=YAKEDU67A2K4MVX6V24YSMV7P4EBQ2PISAHWHSYBROV547NNETZA, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.738895984 (128h1m4.719248626s) [noise=true , address=77.90.31.14:4664 , piece_id=JDJQWUZ7UVHH2NOJFWZYEC5PMPSGODDSOC4HL2A6HCX5Y5GJ6QTQ, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.739265227 (128h1m4.719617863s) [noise=true , address=77.90.31.11:4559 , piece_id=BKPMAM6WXNAO7DFAIBV57UHK5VOFZ5CTIUR2ICUOD4BLIO7PEEHQ, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.739818214 (128h1m4.720170854s) [noise=true , address=77.90.31.18:4830 , piece_id=N7WVZJNG37OLFZKI7ZSJHY5JQYGVJKDNW4SX4M352PSYNL6DFQVQ, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:24.740338221 (128h1m4.720690857s) [noise=true , address=45.91.116.12:4231 , piece_id=24EP2VB3HILY65CWDUUNZA3YVZGVJDRRNL5FLUFOSFMFTPKIAHMA, seg_pos={0 0}, upload=112529]: Uploading piece... Dec 10 23:29:25.306195857 (128h1m5.286548589s) [upload=112529]: Next returned nil segment Dec 10 23:29:25.306217326 (128h1m5.286569972s) [upload=112529]: Waiting for error group managing segments... Dec 10 23:29:25.472552141 (128h1m5.452904791s) [noise=true , address=45.91.116.12:4231 , piece_id=24EP2VB3HILY65CWDUUNZA3YVZGVJDRRNL5FLUFOSFMFTPKIAHMA, seg_pos={0 0}, upload=112529]: Done uploading piece. err: <nil> Dec 10 23:29:25.473330077 (128h1m5.453682728s) [noise=true , address=145.79.227.48:4329 , piece_id=PWR6FMYUQ6M3NR34DOZ2KXJ2BR7Z7Z2NL5GYCDMC7MM4JSQ4ISVQ, seg_pos={0 0}, upload=112529]: Done uploading piece. err: <nil> Dec 10 23:29:25.474078985 (128h1m5.45443167s) [noise=true , address=45.91.116.4:4959 , piece_id=BMN7VCKIYSNRFCBXUF7H6R2LBLD4IT3SQMQX4RDGNXOOURYGPBPA, seg_pos={0 0}, upload=112529]: Done uploading piece. err: <nil> Dec 10 23:29:25.474866536 (128h1m5.455219196s) [noise=true , address=145.79.227.28:4757 , piece_id=AEZSB5ELGJD2R7EC37PEIYTUXFX6RZH6TRNR46OKMVNFU5P4IVZA, seg_pos={0 0}, upload=112529]: Done uploading piece. err: <nil> Dec 10 23:29:25.475558779 (128h1m5.455911422s) [noise=true , address=45.91.116.10:4091 , piece_id=YAKEDU67A2K4MVX6V24YSMV7P4EBQ2PISAHWHSYBROV547NNETZA, seg_pos={0 0}, upload=112529]: Done uploading piece. err: <nil> Dec 10 23:29:25.491520202 (128h1m5.471872841s) [noise=true , address=145.79.227.25:4646 , piece_id=EHCBJMRGQONG66IBBLHQGMMHUSVBRDPGS3GSYFAETO3EXZRSRLGA, seg_pos={0 0}, upload=112529]: Done uploading piece. err: <nil> Dec 10 23:29:25.502741083 (128h1m5.483093721s) [noise=true , address=145.79.227.29:4957 , piece_id=2YBLZFDAPOTVZRTDN662JTXKVPB44YITDJZN4T5SKEZ4HHACJKTQ, seg_pos={0 0}, upload=112529]: Done uploading piece. err: <nil> Dec 10 23:29:25.503687707 (128h1m5.484040351s) [noise=true , address=45.91.116.5:4312 , piece_id=6ETOURPL3VSJ2HKQN4DQEIZG35YPAMD7FPITAA6X4NYJG5LV2PWQ, seg_pos={0 0}, upload=112529]: Done uploading piece. err: <nil> Dec 10 23:29:26.451758203 (128h1m6.432110839s) [noise=true , address=145.79.227.44:4201 , piece_id=M4NOMVMGD4H7SYKK4TFOPBEFWXBFQ2R527FGC5ZCF2DN4BS7QQAA, seg_pos={0 0}, upload=112529]: Done uploading piece. err: <nil> Dec 10 23:29:26.456268679 (128h1m6.436621303s) [noise=true , address=145.79.227.30:4778 , piece_id=MEC3V2IROPODBRVZWI6RVTPNZOYHXZIOKTXRHULWU56XBI6YYJ7Q, seg_pos={0 0}, upload=112529]: Done uploading piece. err: <nil> Dec 10 23:29:36.530811506 (128h1m16.511164142s) [noise=true , address=64.49.13.169:4715 , piece_id=QD4ENFOYMQJD2BNH47M5YX2SNYCD72FENVZUJKSTKXFLOBMMVHPQ, seg_pos={0 0}, upload=112529]: Done uploading piece. err: <nil> Dec 10 23:30:24.297723000 (128h2m4.278075647s) [noise=true , address=77.90.31.16:4765 , piece_id=NRSFGLZJDRW7UNYNY6TGRVMB3XFHJ2N3OUC7FF2KR6Y4SNTB35JQ, seg_pos={0 0}, upload=112529]: Done uploading piece. err: ecclient: upload failed (node:12CSrtv2fMmFBx9zje9zBevpVzxqyhH7fL7VnHbBFEYFRPuwCKK, address:77.90.31.16:4765): protocol: rpc: dial tcp 77.90.31.16:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:24.297748493 (128h2m4.278101138s) [noise=true , address=77.90.31.21:4906 , piece_id=HKZLIRO4CUMPOHO3G75JW3IXWVQ2VZXDUOC3YI7L4FXZ3PGJDLDA, seg_pos={0 0}, upload=112529]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: write tcp 139.59.144.22:38610->77.90.31.21:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:24.297974322 (128h2m4.278326969s) [noise=true , address=77.90.31.18:4830 , piece_id=N7WVZJNG37OLFZKI7ZSJHY5JQYGVJKDNW4SX4M352PSYNL6DFQVQ, seg_pos={0 0}, upload=112529]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 139.59.144.22:46634->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:24.298026916 (128h2m4.278379555s) [noise=true , address=77.90.31.13:4630 , piece_id=OKZUCQYP5TXZTUTEJQ5Y3L3NMRAP5W2TXWZ5TN4DXKMF5HLFH6FA, seg_pos={0 0}, upload=112529]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: write tcp 139.59.144.22:41594->77.90.31.13:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:24.298114245 (128h2m4.278466872s) [noise=true , address=77.90.31.14:4664 , piece_id=JDJQWUZ7UVHH2NOJFWZYEC5PMPSGODDSOC4HL2A6HCX5Y5GJ6QTQ, seg_pos={0 0}, upload=112529]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: write tcp 139.59.144.22:49234->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:24.298137479 (128h2m4.278490125s) [noise=true , address=77.90.31.12:4600 , piece_id=Y6K6SXYFBF66TLPTG3FKOU57DPMKU3KBKATSLWCLWMN7YYDU4H5Q, seg_pos={0 0}, upload=112529]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:24.298288608 (128h2m4.278641251s) [noise=true , address=77.90.31.11:4559 , piece_id=BKPMAM6WXNAO7DFAIBV57UHK5VOFZ5CTIUR2ICUOD4BLIO7PEEHQ, seg_pos={0 0}, upload=112529]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 139.59.144.22:53834->77.90.31.11:4559: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:24.298315472 (128h2m4.278668099s) [seg_pos={0 0}, upload=112529]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:30:24.414231544 (128h2m4.394584184s) [upload=112529]: Done uploading segments... Dec 10 23:30:24.414265520 (128h2m4.394618161s) [upload=112529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.119799754s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................................................XXX......] duration: 2.914526181s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:30:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.914550909s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)