Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [.........................X..................X.....XXXXXX.XXX] duration: 30.132296618s FAIL (2025-07-26 23:13:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.13232143s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................X.XX...X] duration: 30.001483789s FAIL (2025-07-26 23:13:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001532241s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.X...X.XX...X..X.XX......X.X.......X.......X........XX....X.] duration: 16.566413522s FAIL (2025-07-26 23:12:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.566433762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............................................X......XXXXXXXX] duration: 30.118074412s FAIL (2025-07-26 23:12:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 26 23:12:16.077176281 (59h36m47.267474862s) [upload=57170]: Done starting upload Jul 26 23:12:16.077315516 (59h36m47.267614096s) [upload=57170]: Uploading segments... Jul 26 23:12:16.077592788 (59h36m47.267891368s) [upload=57170]: Got next segment. Inline: false Jul 26 23:12:16.077632646 (59h36m47.267931226s) [upload=57170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:12:16.077637838 (59h36m47.267936416s) [upload=57170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:12:16.188884022 (59h36m47.379182591s) [seg_pos={0 0}, upload=57170]: Begin upload segment... Jul 26 23:12:16.188934571 (59h36m47.379233144s) [seg_pos={0 0}, upload=57170]: Done begin upload segment. Jul 26 23:12:16.188971657 (59h36m47.379270239s) [noise=true , address=104.36.82.3:4765 , piece_id=YK6HGHNBPDXVJC27TZBG2UDNPWYAPKBRI5TTM3QGJ4GUGHEJ4FGA, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.188988904 (59h36m47.379287486s) [noise=true , address=104.36.82.14:4222 , piece_id=YN36JN7AB4JRAQKF6ZJVWQWZXXDJOQY63HBW5HTV4IOOJ5PORHOQ, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.189360242 (59h36m47.379658823s) [noise=true , address=104.36.82.6:4873 , piece_id=B3P7CG7DCYWD5PNMI74SRFQDEAH74GGCFY2EEX5JWTX2QLA4PFGA, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.189467869 (59h36m47.379766467s) [noise=true , address=45.91.116.10:4082 , piece_id=XQDLQ4GU4IKA5YWS2FD54Z45T74PTL6RDML7MKIFDY227MV4VW7Q, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.189740287 (59h36m47.380038875s) [noise=true , address=104.36.82.2:4729 , piece_id=6NHODCBGWOQJBMLAAWRCCHYUZFC7IQ4K3YIB5HFFRCMBL2XJ55BA, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.189965484 (59h36m47.380264066s) [noise=true , address=104.36.82.6:4875 , piece_id=TVXSBAMRGLM5WV4VEERM5WZSWX2F4DL5ZH67MICJQTLPO4WDFZFA, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.190137656 (59h36m47.380436236s) [noise=true , address=104.36.82.11:4115 , piece_id=DHJMYXAJTYOAHLJKL5B6DWGBS5ATCAQQFW4CASDG6NRXRP6KY2MQ, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.190439756 (59h36m47.380738343s) [noise=true , address=104.36.82.12:4151 , piece_id=7KMSZ7XRVODEE3BXW5VLX7T5LQXLEMD6Q6O63BTKBSBS4QYDCXXA, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.190738852 (59h36m47.381037434s) [noise=true , address=45.91.116.8:4045 , piece_id=B3ID3XH4YMZSI4QCYIVS7JMIBETFCCRDCXITNYTBXDOOXFUW45YA, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.191035970 (59h36m47.381334549s) [noise=true , address=45.91.116.11:4118 , piece_id=FWQXVHXFL5MIXYGISHXBILOK7VDPHDSVAOSKPBQ2B4VDLVWOYEHQ, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.191351276 (59h36m47.381649855s) [noise=true , address=104.36.82.4:4804 , piece_id=OEZBOUSUTDWMNGL4OFCBC2GTZPFQ2UC5ITMJOE6HBMPCNCGIJ4HQ, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.191652970 (59h36m47.381951552s) [noise=true , address=104.36.82.3:4767 , piece_id=WHH3AIJ3J2QMTV6CP27JHZMJBECCZLM3XO52UU4OZ565RTOWXIRQ, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.191755205 (59h36m47.382053903s) [noise=true , address=77.90.31.14:4663 , piece_id=XU4UPKLGMMLMGVJ5JI64BWFZY36UR3MO6EUEOIQT5ORRFX3RSO7A, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.192190010 (59h36m47.382488604s) [noise=true , address=45.91.116.10:4081 , piece_id=5JLT5JJZDBEYI6KVQADRZ3JY3U2TPK3AGCBMDXLPETQ3HIOL65DQ, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.192678586 (59h36m47.382977175s) [noise=true , address=77.90.31.12:4590 , piece_id=UPAHFXHYWLTRTLP6VXFDFLITSLVAWSZLRRLQLP6DAWUGW6DYHGVQ, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.193606581 (59h36m47.383905165s) [noise=true , address=45.91.116.7:4011 , piece_id=ZFWJN4VC6KHTCP6IVZQY4W5TCGNT4WRX2BLGPFM4RFIUJIJIQ2CA, seg_pos={0 0}, upload=57170]: Uploading piece... Jul 26 23:12:16.818464589 (59h36m48.008763177s) [upload=57170]: Next returned nil segment Jul 26 23:12:16.818486755 (59h36m48.008785344s) [upload=57170]: Waiting for error group managing segments... Jul 26 23:12:16.953630897 (59h36m48.143929478s) [noise=true , address=45.91.116.7:4011 , piece_id=ZFWJN4VC6KHTCP6IVZQY4W5TCGNT4WRX2BLGPFM4RFIUJIJIQ2CA, seg_pos={0 0}, upload=57170]: Done uploading piece. err: <nil> Jul 26 23:12:16.953788690 (59h36m48.144087262s) [noise=true , address=45.91.116.10:4081 , piece_id=5JLT5JJZDBEYI6KVQADRZ3JY3U2TPK3AGCBMDXLPETQ3HIOL65DQ, seg_pos={0 0}, upload=57170]: Done uploading piece. err: <nil> Jul 26 23:12:16.955866084 (59h36m48.146164663s) [noise=true , address=45.91.116.8:4045 , piece_id=B3ID3XH4YMZSI4QCYIVS7JMIBETFCCRDCXITNYTBXDOOXFUW45YA, seg_pos={0 0}, upload=57170]: Done uploading piece. err: <nil> Jul 26 23:12:16.957376146 (59h36m48.147674718s) [noise=true , address=45.91.116.10:4082 , piece_id=XQDLQ4GU4IKA5YWS2FD54Z45T74PTL6RDML7MKIFDY227MV4VW7Q, seg_pos={0 0}, upload=57170]: Done uploading piece. err: <nil> Jul 26 23:12:16.960309239 (59h36m48.150607816s) [noise=true , address=45.91.116.11:4118 , piece_id=FWQXVHXFL5MIXYGISHXBILOK7VDPHDSVAOSKPBQ2B4VDLVWOYEHQ, seg_pos={0 0}, upload=57170]: Done uploading piece. err: <nil> Jul 26 23:12:16.964708134 (59h36m48.155006722s) [noise=true , address=77.90.31.14:4663 , piece_id=XU4UPKLGMMLMGVJ5JI64BWFZY36UR3MO6EUEOIQT5ORRFX3RSO7A, seg_pos={0 0}, upload=57170]: Done uploading piece. err: <nil> Jul 26 23:12:29.816710573 (59h37m1.007009144s) [noise=true , address=104.36.82.11:4115 , piece_id=DHJMYXAJTYOAHLJKL5B6DWGBS5ATCAQQFW4CASDG6NRXRP6KY2MQ, seg_pos={0 0}, upload=57170]: Done uploading piece. err: <nil> Jul 26 23:12:38.495375845 (59h37m9.685674417s) [noise=true , address=104.36.82.3:4765 , piece_id=YK6HGHNBPDXVJC27TZBG2UDNPWYAPKBRI5TTM3QGJ4GUGHEJ4FGA, seg_pos={0 0}, upload=57170]: Done uploading piece. err: <nil> Jul 26 23:12:40.236022833 (59h37m11.426321411s) [noise=true , address=104.36.82.12:4151 , piece_id=7KMSZ7XRVODEE3BXW5VLX7T5LQXLEMD6Q6O63BTKBSBS4QYDCXXA, seg_pos={0 0}, upload=57170]: Done uploading piece. err: <nil> Jul 26 23:12:45.800306768 (59h37m16.990605347s) [noise=true , address=104.36.82.6:4873 , piece_id=B3P7CG7DCYWD5PNMI74SRFQDEAH74GGCFY2EEX5JWTX2QLA4PFGA, seg_pos={0 0}, upload=57170]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:12:45.800665379 (59h37m16.990963958s) [noise=true , address=104.36.82.6:4875 , piece_id=TVXSBAMRGLM5WV4VEERM5WZSWX2F4DL5ZH67MICJQTLPO4WDFZFA, seg_pos={0 0}, upload=57170]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 139.59.144.22:48156->104.36.82.6:4875: use of closed network connection; write tcp 139.59.144.22:48156->104.36.82.6:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48156->104.36.82.6:4875: use of closed network connection Jul 26 23:12:45.800724694 (59h37m16.991023273s) [noise=true , address=104.36.82.4:4804 , piece_id=OEZBOUSUTDWMNGL4OFCBC2GTZPFQ2UC5ITMJOE6HBMPCNCGIJ4HQ, seg_pos={0 0}, upload=57170]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:12:45.800768189 (59h37m16.991066769s) [noise=true , address=104.36.82.2:4729 , piece_id=6NHODCBGWOQJBMLAAWRCCHYUZFC7IQ4K3YIB5HFFRCMBL2XJ55BA, seg_pos={0 0}, upload=57170]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:12:45.800815025 (59h37m16.991113605s) [noise=true , address=104.36.82.14:4222 , piece_id=YN36JN7AB4JRAQKF6ZJVWQWZXXDJOQY63HBW5HTV4IOOJ5PORHOQ, seg_pos={0 0}, upload=57170]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:55384->104.36.82.14:4222: use of closed network connection; write tcp 139.59.144.22:55384->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55384->104.36.82.14:4222: use of closed network connection Jul 26 23:12:45.800844591 (59h37m16.991143163s) [noise=true , address=77.90.31.12:4590 , piece_id=UPAHFXHYWLTRTLP6VXFDFLITSLVAWSZLRRLQLP6DAWUGW6DYHGVQ, seg_pos={0 0}, upload=57170]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:37432->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:45.800859639 (59h37m16.991158255s) [noise=true , address=104.36.82.3:4767 , piece_id=WHH3AIJ3J2QMTV6CP27JHZMJBECCZLM3XO52UU4OZ565RTOWXIRQ, seg_pos={0 0}, upload=57170]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:46390->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:46390->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46390->104.36.82.3:4767: use of closed network connection Jul 26 23:12:45.800880633 (59h37m16.991179237s) [seg_pos={0 0}, upload=57170]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:12:45.916031410 (59h37m17.106329984s) [upload=57170]: Done uploading segments... Jul 26 23:12:45.916074901 (59h37m17.106373483s) [upload=57170]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.118099288s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..X...X.XX...X..X.XX......X.X.......X.......X........XX....X] duration: 1m0.00199747s FAIL (2025-07-26 23:12:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002025949s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.XX..X.] duration: 52.244198332s FAIL (2025-07-26 23:12:20 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0946594s) [upload=57142]: Starting upload Jul 26 23:10:27.460731738 (59h34m58.651030327s) [upload=57142]: Done starting upload Jul 26 23:10:27.460909600 (59h34m58.651208167s) [upload=57142]: Uploading segments... Jul 26 23:10:27.462056974 (59h34m58.652355549s) [upload=57142]: Got next segment. Inline: false Jul 26 23:10:27.462123048 (59h34m58.652421631s) [upload=57142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:10:27.462131081 (59h34m58.652429667s) [upload=57142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:10:27.659395008 (59h34m58.849693596s) [seg_pos={0 0}, upload=57142]: Begin upload segment... Jul 26 23:10:27.659486666 (59h34m58.849785253s) [seg_pos={0 0}, upload=57142]: Done begin upload segment. Jul 26 23:10:27.659525498 (59h34m58.849824092s) [noise=true , address=45.91.116.5:4310 , piece_id=HC2QMH2V3A7LGLRB3RFPDA244VWBNCYDAU5BO62VQLB7DTAQXNDQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.659537846 (59h34m58.849836439s) [noise=true , address=156.146.43.225:4449 , piece_id=UXM2SMXVLGQPUNOVSI62PCFHT3PR4YUTV2JISAWFZNUJHKVDNRUQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.659985291 (59h34m58.850283874s) [noise=true , address=104.36.82.47:4532 , piece_id=QZ45QSEDTLDOX7KFZKJVOMQG2MEMKI2FBFRL66H5GLP6P45TX55A, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.660071744 (59h34m58.850370334s) [noise=true , address=104.36.82.56:4248 , piece_id=LMTW5UWTSH2X2VVD76SI4Z625DDPKJITAXDSGTVRPYS6UCICBX7Q, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.660706104 (59h34m58.851004693s) [noise=true , address=77.90.31.18:4823 , piece_id=EBYCZEE2VIVJ6WDKMJ6RE6DTXFTWKP3NYK23TMTZ3TCXSU3Y5OMQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.662360875 (59h34m58.852659465s) [noise=true , address=79.127.205.245:4250 , piece_id=7ZS7CEUGUSQZWX6UN6FMYUQTBF23OBNI5PEM3H2VK4JCMKLJF5MA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.662779228 (59h34m58.853077813s) [noise=true , address=77.90.31.11:4575 , piece_id=2WRRRFS2TWWH6RY3EYAEL4GXIKSJTCESOWUVTFSPCNFLOL2JXTVQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.663175050 (59h34m58.853473637s) [noise=true , address=104.36.82.54:4180 , piece_id=R7HUA4JHNHHNTZK2HYVEIRHZWVVTJ4TCUJLXPDESMOVFO33E6UGA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.663575967 (59h34m58.853874561s) [noise=true , address=104.36.82.46:4499 , piece_id=LMVELJMAGNSYUEXBPBLKELAFY4NHIMKJDLZAQALBQSRLYGUDMG4Q, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.664018898 (59h34m58.854317483s) [noise=true , address=77.90.31.13:4644 , piece_id=A2SL7EGOGFY5ZNBXK37S7GAPYO2ZVGONZBSMXMCADQGDMTPVJXAA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.664582865 (59h34m58.854881456s) [noise=true , address=79.127.219.38:4006 , piece_id=FIRB5BTFFXTZMHF4CM7WZQCTYJUSGF5EW5JNYOL7JWOFGHDMDUIA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.664990405 (59h34m58.85528899s) [noise=true , address=79.127.201.215:4341 , piece_id=ACJWOX3MHSMCOLZTK46RSIESSAXRMU4Q6DVG7RSP4GERPALLCTHA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.665295783 (59h34m58.855594369s) [noise=true , address=79.127.220.101:4048 , piece_id=VZZK23NT3BLIOS7LVIJL2F66FBQ4SBEGU5U2EE24HY2J6YLWQMKA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.665673903 (59h34m58.855972495s) [noise=true , address=104.36.82.55:4193 , piece_id=G2SMTRFASL7CTTMCLRZCSX5NUW35EZKWMBYENWDJYTMHJSLU6I4A, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.666167731 (59h34m58.856466319s) [noise=true , address=104.36.82.53:4150 , piece_id=43YNB74RRIEB2W2H2THJCROU3BVKNCEUWWRP57LKWKDVYY6MS2EA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.666532785 (59h34m58.856831373s) [noise=true , address=79.127.205.234:4589 , piece_id=PHOVPCQ5YRK2ACKFQ2O2ZJ4BHC6L7NF5NYREDUAXHYCCVZYCDTUA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.666808055 (59h34m58.857106634s) [noise=true , address=77.90.31.21:4906 , piece_id=ZAMXMURAFIOUFC6NYJIWZZ5YSPLY3DDIKAHM3YQ6ZCOGB5OVJOZQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.667703830 (59h34m58.858002423s) [noise=true , address=104.36.82.52:4113 , piece_id=OD265OTPTJDBIAGHNT7GTNSHFMVJEYM65QPCRGCGWRJGRHZL2M5Q, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:28.193572301 (59h34m59.383870887s) [noise=true , address=79.127.220.101:4048 , piece_id=VZZK23NT3BLIOS7LVIJL2F66FBQ4SBEGU5U2EE24HY2J6YLWQMKA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:10:28.229500104 (59h34m59.419798674s) [upload=57142]: Next returned nil segment Jul 26 23:10:28.229522067 (59h34m59.419820647s) [upload=57142]: Waiting for error group managing segments... Jul 26 23:10:28.395706412 (59h34m59.586004979s) [noise=true , address=79.127.219.38:4006 , piece_id=FIRB5BTFFXTZMHF4CM7WZQCTYJUSGF5EW5JNYOL7JWOFGHDMDUIA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:28.425478908 (59h34m59.615777478s) [noise=true , address=77.90.31.11:4575 , piece_id=2WRRRFS2TWWH6RY3EYAEL4GXIKSJTCESOWUVTFSPCNFLOL2JXTVQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:28.466610022 (59h34m59.656908588s) [noise=true , address=77.90.31.18:4823 , piece_id=EBYCZEE2VIVJ6WDKMJ6RE6DTXFTWKP3NYK23TMTZ3TCXSU3Y5OMQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:28.470728977 (59h34m59.66102756s) [noise=true , address=77.90.31.21:4906 , piece_id=ZAMXMURAFIOUFC6NYJIWZZ5YSPLY3DDIKAHM3YQ6ZCOGB5OVJOZQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:28.678953845 (59h34m59.869252417s) [noise=true , address=79.127.201.215:4341 , piece_id=ACJWOX3MHSMCOLZTK46RSIESSAXRMU4Q6DVG7RSP4GERPALLCTHA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:28.964526833 (59h35m0.154825405s) [noise=true , address=79.127.205.234:4589 , piece_id=PHOVPCQ5YRK2ACKFQ2O2ZJ4BHC6L7NF5NYREDUAXHYCCVZYCDTUA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:28.987048012 (59h35m0.177346584s) [noise=true , address=79.127.205.245:4250 , piece_id=7ZS7CEUGUSQZWX6UN6FMYUQTBF23OBNI5PEM3H2VK4JCMKLJF5MA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:30.430481439 (59h35m1.62078003s) [noise=true , address=156.146.43.225:4449 , piece_id=UXM2SMXVLGQPUNOVSI62PCFHT3PR4YUTV2JISAWFZNUJHKVDNRUQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:33.897370788 (59h35m5.087669358s) [noise=true , address=45.91.116.5:4310 , piece_id=HC2QMH2V3A7LGLRB3RFPDA244VWBNCYDAU5BO62VQLB7DTAQXNDQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:35.234858118 (59h35m6.425156692s) [noise=true , address=104.36.82.55:4193 , piece_id=G2SMTRFASL7CTTMCLRZCSX5NUW35EZKWMBYENWDJYTMHJSLU6I4A, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:11:12.969867685 (59h35m44.160166257s) [noise=true , address=104.36.82.52:4113 , piece_id=OD265OTPTJDBIAGHNT7GTNSHFMVJEYM65QPCRGCGWRJGRHZL2M5Q, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:11:13.989375348 (59h35m45.179673917s) [noise=true , address=104.36.82.56:4248 , piece_id=LMTW5UWTSH2X2VVD76SI4Z625DDPKJITAXDSGTVRPYS6UCICBX7Q, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:11:19.843947729 (59h35m51.034246298s) [noise=true , address=104.36.82.54:4180 , piece_id=R7HUA4JHNHHNTZK2HYVEIRHZWVVTJ4TCUJLXPDESMOVFO33E6UGA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:11:26.323943181 (59h35m57.514241748s) [noise=true , address=104.36.82.53:4150 , piece_id=43YNB74RRIEB2W2H2THJCROU3BVKNCEUWWRP57LKWKDVYY6MS2EA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:11:26.323968911 (59h35m57.514267503s) [seg_pos={0 0}, upload=57142]: Segment reached optimal threshold of 14 pieces. Jul 26 23:11:26.324097706 (59h35m57.514396276s) [noise=true , address=104.36.82.46:4499 , piece_id=LMVELJMAGNSYUEXBPBLKELAFY4NHIMKJDLZAQALBQSRLYGUDMG4Q, seg_pos={0 0}, upload=57142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc): protocol: expected piece hash; context canceled; EOF Jul 26 23:11:26.324886152 (59h35m57.515184722s) [noise=true , address=104.36.82.47:4532 , piece_id=QZ45QSEDTLDOX7KFZKJVOMQG2MEMKI2FBFRL66H5GLP6P45TX55A, seg_pos={0 0}, upload=57142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL): protocol: expected piece hash; context canceled; EOF Jul 26 23:11:26.325527404 (59h35m57.515825982s) [noise=true , address=77.90.31.13:4644 , piece_id=A2SL7EGOGFY5ZNBXK37S7GAPYO2ZVGONZBSMXMCADQGDMTPVJXAA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h): protocol: rpc: dial tcp 77.90.31.13:4644: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:26.325566786 (59h35m57.515865373s) [seg_pos={0 0}, upload=57142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 26 23:11:26.325647962 (59h35m57.515946552s) [upload=57142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:11:26.325654211 (59h35m57.515952801s) [upload=57142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:11:26.481578936 (59h35m57.671877516s) [upload=57142]: Done uploading segments... Jul 26 23:11:26.481598726 (59h35m57.671897296s) [upload=57142]: Upload finished. err: <nil> Duration: 52.24431918s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........................X..................X.....XXXXXX.XX] duration: 30.212149825s FAIL (2025-07-26 23:12:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.212177625s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X.XX...] duration: 20.503646403s FAIL (2025-07-26 23:12:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.503751864s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........................X..................X.....XXXXXX.X] duration: 30.111844587s FAIL (2025-07-26 23:11:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.111868094s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............................................X......XXXXXXX] duration: 30.109990334s FAIL (2025-07-26 23:11:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 23:10:48.869081778 (59h35m20.059380367s) [upload=57141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:10:48.869088819 (59h35m20.059387406s) [upload=57141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:10:49.135006620 (59h35m20.325305195s) [upload=57141]: Done uploading segments... Jul 26 23:10:49.135055934 (59h35m20.325354514s) [upload=57141]: Upload finished. err: metaclient: context deadline exceeded Jul 26 23:11:17.536278895 (59h35m48.726577477s) [upload=57155]: Starting upload Jul 26 23:11:17.536367880 (59h35m48.726666447s) [upload=57155]: Done starting upload Jul 26 23:11:17.536557698 (59h35m48.726856271s) [upload=57155]: Uploading segments... Jul 26 23:11:17.536844648 (59h35m48.727143229s) [upload=57155]: Got next segment. Inline: false Jul 26 23:11:17.536910085 (59h35m48.727208668s) [upload=57155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:11:17.536917722 (59h35m48.727216303s) [upload=57155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:11:17.659773370 (59h35m48.850071953s) [seg_pos={0 0}, upload=57155]: Begin upload segment... Jul 26 23:11:17.659853664 (59h35m48.850152249s) [noise=true , address=104.36.82.6:4873 , piece_id=XH7YFD6DG5UGGCHDTSNDNLHWIDCTY6AS3K6O6QV2VJT77A2SAGGA, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.660356128 (59h35m48.850654718s) [noise=true , address=104.36.82.16:4295 , piece_id=SGS2UEE4VHYBE5DRBU5QELP4E2FWVXGH5IGKDIVJ2NE7KLG66PVA, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.661146571 (59h35m48.85144514s) [seg_pos={0 0}, upload=57155]: Done begin upload segment. Jul 26 23:11:17.661182194 (59h35m48.851480773s) [noise=true , address=104.36.82.8:4971 , piece_id=HD3EK566G6IOQLN767VUGDJZ7DAVWCCG6VXCLRL5LQI2YJCCXBYQ, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.661785023 (59h35m48.852083611s) [noise=true , address=45.91.116.4:4937 , piece_id=H5CFUU72VXNBDYERM4J5CXDUSIWN5MYZKL6TCVN75PY4REP7TXIA, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.662220254 (59h35m48.852518841s) [noise=true , address=77.90.31.13:4626 , piece_id=WDJLTXD2OXE6W2KDIF5A3UINEMQBWDEH5SWBB7DWJZWSSJY5S3KA, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.662486865 (59h35m48.852785452s) [noise=true , address=45.91.116.7:4009 , piece_id=ILKZKZDPIBBW3C6ZQ72JA6O24Q6SMJBLUFGAWKNBDDUP3MWLEDBQ, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.662576089 (59h35m48.852874674s) [noise=true , address=104.36.82.4:4803 , piece_id=WKVD5CRAN6VK2RTBWFJFDALB7U576BU3URVDDJEULV4Q2LO3PFDA, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.662844653 (59h35m48.853143238s) [noise=true , address=104.36.82.9:4005 , piece_id=JJO7X6RL3XFNOTVT6C6K4HJXUBCOITVFSZ7XTJXFFFWOHRSUFXOA, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.663170854 (59h35m48.853469435s) [noise=true , address=45.91.116.8:4048 , piece_id=W6OFOKX4NMGEAFOX3253H6UGU7BGSQ2ZQWDF5VWYJJYRUENQWDQA, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.663188607 (59h35m48.853487196s) [noise=true , address=45.91.116.12:4227 , piece_id=BSCTUJMKZHXWBYLKVM3ZXKLID4KTQREFQBTQAH7L3VOHZBCQUDJA, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.663460073 (59h35m48.853758652s) [noise=true , address=104.36.82.3:4767 , piece_id=2JK3MPC4ZI4PYPBHHD7A2O72VZJ4G5O27W55R2EPARCXV3BS37BQ, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.663674832 (59h35m48.853973419s) [noise=true , address=104.36.82.9:4006 , piece_id=6YKIFXLZ5KICK6ITJ3Y7RYOFT2K5BU64RG4WXHWG4OW66XAKS5HQ, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.663763540 (59h35m48.854062122s) [noise=true , address=45.91.116.11:4118 , piece_id=ZAGLXIQI22TFKFGT3Z2GDZPJVHQVAV67D2PGM7HUUJIOKKF5B7KQ, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.664042829 (59h35m48.854341414s) [noise=true , address=77.90.31.14:4663 , piece_id=DXSHYQBVKT37TTGB3YIABCGW5YOFJZLQ3IBKMJTSO4UC6QMDAWZQ, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.664245671 (59h35m48.854544261s) [noise=true , address=104.36.82.15:4257 , piece_id=YMFKAWFTXETL5JB45JVEPBHPLV4KAAKHVOIONOIEVYKLWICBUIMA, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:17.664736621 (59h35m48.855035211s) [noise=true , address=104.36.82.2:4731 , piece_id=YL6O3UKXLH3FMCOXF3QRLMDLKH3RGOBBEVADBLIUXUKUJRTN3O3A, seg_pos={0 0}, upload=57155]: Uploading piece... Jul 26 23:11:18.283210257 (59h35m49.473508825s) [upload=57155]: Next returned nil segment Jul 26 23:11:18.283233057 (59h35m49.473531647s) [upload=57155]: Waiting for error group managing segments... Jul 26 23:11:18.419256547 (59h35m49.60955512s) [noise=true , address=45.91.116.11:4118 , piece_id=ZAGLXIQI22TFKFGT3Z2GDZPJVHQVAV67D2PGM7HUUJIOKKF5B7KQ, seg_pos={0 0}, upload=57155]: Done uploading piece. err: <nil> Jul 26 23:11:18.431199790 (59h35m49.621498361s) [noise=true , address=104.36.82.16:4295 , piece_id=SGS2UEE4VHYBE5DRBU5QELP4E2FWVXGH5IGKDIVJ2NE7KLG66PVA, seg_pos={0 0}, upload=57155]: Done uploading piece. err: <nil> Jul 26 23:11:18.432661968 (59h35m49.622960533s) [noise=true , address=77.90.31.14:4663 , piece_id=DXSHYQBVKT37TTGB3YIABCGW5YOFJZLQ3IBKMJTSO4UC6QMDAWZQ, seg_pos={0 0}, upload=57155]: Done uploading piece. err: <nil> Jul 26 23:11:18.433048445 (59h35m49.623347012s) [noise=true , address=45.91.116.4:4937 , piece_id=H5CFUU72VXNBDYERM4J5CXDUSIWN5MYZKL6TCVN75PY4REP7TXIA, seg_pos={0 0}, upload=57155]: Done uploading piece. err: <nil> Jul 26 23:11:18.433121619 (59h35m49.623420185s) [noise=true , address=45.91.116.8:4048 , piece_id=W6OFOKX4NMGEAFOX3253H6UGU7BGSQ2ZQWDF5VWYJJYRUENQWDQA, seg_pos={0 0}, upload=57155]: Done uploading piece. err: <nil> Jul 26 23:11:18.434073732 (59h35m49.624372306s) [noise=true , address=45.91.116.12:4227 , piece_id=BSCTUJMKZHXWBYLKVM3ZXKLID4KTQREFQBTQAH7L3VOHZBCQUDJA, seg_pos={0 0}, upload=57155]: Done uploading piece. err: <nil> Jul 26 23:11:18.500314846 (59h35m49.69061342s) [noise=true , address=45.91.116.7:4009 , piece_id=ILKZKZDPIBBW3C6ZQ72JA6O24Q6SMJBLUFGAWKNBDDUP3MWLEDBQ, seg_pos={0 0}, upload=57155]: Done uploading piece. err: <nil> Jul 26 23:11:23.880127477 (59h35m55.07042605s) [noise=true , address=104.36.82.8:4971 , piece_id=HD3EK566G6IOQLN767VUGDJZ7DAVWCCG6VXCLRL5LQI2YJCCXBYQ, seg_pos={0 0}, upload=57155]: Done uploading piece. err: <nil> Jul 26 23:11:29.827803326 (59h36m1.018101897s) [noise=true , address=104.36.82.3:4767 , piece_id=2JK3MPC4ZI4PYPBHHD7A2O72VZJ4G5O27W55R2EPARCXV3BS37BQ, seg_pos={0 0}, upload=57155]: Done uploading piece. err: <nil> Jul 26 23:11:39.236648222 (59h36m10.426946813s) [noise=true , address=104.36.82.2:4731 , piece_id=YL6O3UKXLH3FMCOXF3QRLMDLKH3RGOBBEVADBLIUXUKUJRTN3O3A, seg_pos={0 0}, upload=57155]: Done uploading piece. err: <nil> Jul 26 23:11:45.008256706 (59h36m16.198555278s) [noise=true , address=104.36.82.6:4873 , piece_id=XH7YFD6DG5UGGCHDTSNDNLHWIDCTY6AS3K6O6QV2VJT77A2SAGGA, seg_pos={0 0}, upload=57155]: Done uploading piece. err: <nil> Jul 26 23:11:47.265475056 (59h36m18.455773645s) [noise=true , address=104.36.82.15:4257 , piece_id=YMFKAWFTXETL5JB45JVEPBHPLV4KAAKHVOIONOIEVYKLWICBUIMA, seg_pos={0 0}, upload=57155]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:11:47.265601507 (59h36m18.455900074s) [noise=true , address=104.36.82.9:4006 , piece_id=6YKIFXLZ5KICK6ITJ3Y7RYOFT2K5BU64RG4WXHWG4OW66XAKS5HQ, seg_pos={0 0}, upload=57155]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 139.59.144.22:38038->104.36.82.9:4006: use of closed network connection; write tcp 139.59.144.22:38038->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38038->104.36.82.9:4006: use of closed network connection Jul 26 23:11:47.265709759 (59h36m18.456008332s) [noise=true , address=104.36.82.4:4803 , piece_id=WKVD5CRAN6VK2RTBWFJFDALB7U576BU3URVDDJEULV4Q2LO3PFDA, seg_pos={0 0}, upload=57155]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:11:47.265725188 (59h36m18.456023758s) [noise=true , address=104.36.82.9:4005 , piece_id=JJO7X6RL3XFNOTVT6C6K4HJXUBCOITVFSZ7XTJXFFFWOHRSUFXOA, seg_pos={0 0}, upload=57155]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 139.59.144.22:43462->104.36.82.9:4005: use of closed network connection; write tcp 139.59.144.22:43462->104.36.82.9:4005: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43462->104.36.82.9:4005: use of closed network connection Jul 26 23:11:47.265770446 (59h36m18.456069026s) [noise=true , address=77.90.31.13:4626 , piece_id=WDJLTXD2OXE6W2KDIF5A3UINEMQBWDEH5SWBB7DWJZWSSJY5S3KA, seg_pos={0 0}, upload=57155]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:47.265805331 (59h36m18.456103909s) [seg_pos={0 0}, upload=57155]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:11:47.373709671 (59h36m18.564008244s) [upload=57155]: Done uploading segments... Jul 26 23:11:47.373736593 (59h36m18.564035162s) [upload=57155]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.110013455s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [.XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.XX..X] duration: 1m0.0038485s FAIL (2025-07-26 23:11:27 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0946594s) [upload=57142]: Starting upload Jul 26 23:10:27.460731738 (59h34m58.651030327s) [upload=57142]: Done starting upload Jul 26 23:10:27.460909600 (59h34m58.651208167s) [upload=57142]: Uploading segments... Jul 26 23:10:27.462056974 (59h34m58.652355549s) [upload=57142]: Got next segment. Inline: false Jul 26 23:10:27.462123048 (59h34m58.652421631s) [upload=57142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:10:27.462131081 (59h34m58.652429667s) [upload=57142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:10:27.659395008 (59h34m58.849693596s) [seg_pos={0 0}, upload=57142]: Begin upload segment... Jul 26 23:10:27.659486666 (59h34m58.849785253s) [seg_pos={0 0}, upload=57142]: Done begin upload segment. Jul 26 23:10:27.659525498 (59h34m58.849824092s) [noise=true , address=45.91.116.5:4310 , piece_id=HC2QMH2V3A7LGLRB3RFPDA244VWBNCYDAU5BO62VQLB7DTAQXNDQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.659537846 (59h34m58.849836439s) [noise=true , address=156.146.43.225:4449 , piece_id=UXM2SMXVLGQPUNOVSI62PCFHT3PR4YUTV2JISAWFZNUJHKVDNRUQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.659985291 (59h34m58.850283874s) [noise=true , address=104.36.82.47:4532 , piece_id=QZ45QSEDTLDOX7KFZKJVOMQG2MEMKI2FBFRL66H5GLP6P45TX55A, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.660071744 (59h34m58.850370334s) [noise=true , address=104.36.82.56:4248 , piece_id=LMTW5UWTSH2X2VVD76SI4Z625DDPKJITAXDSGTVRPYS6UCICBX7Q, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.660706104 (59h34m58.851004693s) [noise=true , address=77.90.31.18:4823 , piece_id=EBYCZEE2VIVJ6WDKMJ6RE6DTXFTWKP3NYK23TMTZ3TCXSU3Y5OMQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.662360875 (59h34m58.852659465s) [noise=true , address=79.127.205.245:4250 , piece_id=7ZS7CEUGUSQZWX6UN6FMYUQTBF23OBNI5PEM3H2VK4JCMKLJF5MA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.662779228 (59h34m58.853077813s) [noise=true , address=77.90.31.11:4575 , piece_id=2WRRRFS2TWWH6RY3EYAEL4GXIKSJTCESOWUVTFSPCNFLOL2JXTVQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.663175050 (59h34m58.853473637s) [noise=true , address=104.36.82.54:4180 , piece_id=R7HUA4JHNHHNTZK2HYVEIRHZWVVTJ4TCUJLXPDESMOVFO33E6UGA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.663575967 (59h34m58.853874561s) [noise=true , address=104.36.82.46:4499 , piece_id=LMVELJMAGNSYUEXBPBLKELAFY4NHIMKJDLZAQALBQSRLYGUDMG4Q, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.664018898 (59h34m58.854317483s) [noise=true , address=77.90.31.13:4644 , piece_id=A2SL7EGOGFY5ZNBXK37S7GAPYO2ZVGONZBSMXMCADQGDMTPVJXAA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.664582865 (59h34m58.854881456s) [noise=true , address=79.127.219.38:4006 , piece_id=FIRB5BTFFXTZMHF4CM7WZQCTYJUSGF5EW5JNYOL7JWOFGHDMDUIA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.664990405 (59h34m58.85528899s) [noise=true , address=79.127.201.215:4341 , piece_id=ACJWOX3MHSMCOLZTK46RSIESSAXRMU4Q6DVG7RSP4GERPALLCTHA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.665295783 (59h34m58.855594369s) [noise=true , address=79.127.220.101:4048 , piece_id=VZZK23NT3BLIOS7LVIJL2F66FBQ4SBEGU5U2EE24HY2J6YLWQMKA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.665673903 (59h34m58.855972495s) [noise=true , address=104.36.82.55:4193 , piece_id=G2SMTRFASL7CTTMCLRZCSX5NUW35EZKWMBYENWDJYTMHJSLU6I4A, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.666167731 (59h34m58.856466319s) [noise=true , address=104.36.82.53:4150 , piece_id=43YNB74RRIEB2W2H2THJCROU3BVKNCEUWWRP57LKWKDVYY6MS2EA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.666532785 (59h34m58.856831373s) [noise=true , address=79.127.205.234:4589 , piece_id=PHOVPCQ5YRK2ACKFQ2O2ZJ4BHC6L7NF5NYREDUAXHYCCVZYCDTUA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.666808055 (59h34m58.857106634s) [noise=true , address=77.90.31.21:4906 , piece_id=ZAMXMURAFIOUFC6NYJIWZZ5YSPLY3DDIKAHM3YQ6ZCOGB5OVJOZQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:27.667703830 (59h34m58.858002423s) [noise=true , address=104.36.82.52:4113 , piece_id=OD265OTPTJDBIAGHNT7GTNSHFMVJEYM65QPCRGCGWRJGRHZL2M5Q, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:10:28.193572301 (59h34m59.383870887s) [noise=true , address=79.127.220.101:4048 , piece_id=VZZK23NT3BLIOS7LVIJL2F66FBQ4SBEGU5U2EE24HY2J6YLWQMKA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:10:28.229500104 (59h34m59.419798674s) [upload=57142]: Next returned nil segment Jul 26 23:10:28.229522067 (59h34m59.419820647s) [upload=57142]: Waiting for error group managing segments... Jul 26 23:10:28.395706412 (59h34m59.586004979s) [noise=true , address=79.127.219.38:4006 , piece_id=FIRB5BTFFXTZMHF4CM7WZQCTYJUSGF5EW5JNYOL7JWOFGHDMDUIA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:28.425478908 (59h34m59.615777478s) [noise=true , address=77.90.31.11:4575 , piece_id=2WRRRFS2TWWH6RY3EYAEL4GXIKSJTCESOWUVTFSPCNFLOL2JXTVQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:28.466610022 (59h34m59.656908588s) [noise=true , address=77.90.31.18:4823 , piece_id=EBYCZEE2VIVJ6WDKMJ6RE6DTXFTWKP3NYK23TMTZ3TCXSU3Y5OMQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:28.470728977 (59h34m59.66102756s) [noise=true , address=77.90.31.21:4906 , piece_id=ZAMXMURAFIOUFC6NYJIWZZ5YSPLY3DDIKAHM3YQ6ZCOGB5OVJOZQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:28.678953845 (59h34m59.869252417s) [noise=true , address=79.127.201.215:4341 , piece_id=ACJWOX3MHSMCOLZTK46RSIESSAXRMU4Q6DVG7RSP4GERPALLCTHA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:28.964526833 (59h35m0.154825405s) [noise=true , address=79.127.205.234:4589 , piece_id=PHOVPCQ5YRK2ACKFQ2O2ZJ4BHC6L7NF5NYREDUAXHYCCVZYCDTUA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:28.987048012 (59h35m0.177346584s) [noise=true , address=79.127.205.245:4250 , piece_id=7ZS7CEUGUSQZWX6UN6FMYUQTBF23OBNI5PEM3H2VK4JCMKLJF5MA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:30.430481439 (59h35m1.62078003s) [noise=true , address=156.146.43.225:4449 , piece_id=UXM2SMXVLGQPUNOVSI62PCFHT3PR4YUTV2JISAWFZNUJHKVDNRUQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:33.897370788 (59h35m5.087669358s) [noise=true , address=45.91.116.5:4310 , piece_id=HC2QMH2V3A7LGLRB3RFPDA244VWBNCYDAU5BO62VQLB7DTAQXNDQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:35.234858118 (59h35m6.425156692s) [noise=true , address=104.36.82.55:4193 , piece_id=G2SMTRFASL7CTTMCLRZCSX5NUW35EZKWMBYENWDJYTMHJSLU6I4A, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:11:12.969867685 (59h35m44.160166257s) [noise=true , address=104.36.82.52:4113 , piece_id=OD265OTPTJDBIAGHNT7GTNSHFMVJEYM65QPCRGCGWRJGRHZL2M5Q, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:11:13.989375348 (59h35m45.179673917s) [noise=true , address=104.36.82.56:4248 , piece_id=LMTW5UWTSH2X2VVD76SI4Z625DDPKJITAXDSGTVRPYS6UCICBX7Q, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:11:19.843947729 (59h35m51.034246298s) [noise=true , address=104.36.82.54:4180 , piece_id=R7HUA4JHNHHNTZK2HYVEIRHZWVVTJ4TCUJLXPDESMOVFO33E6UGA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:11:26.323943181 (59h35m57.514241748s) [noise=true , address=104.36.82.53:4150 , piece_id=43YNB74RRIEB2W2H2THJCROU3BVKNCEUWWRP57LKWKDVYY6MS2EA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:11:26.323968911 (59h35m57.514267503s) [seg_pos={0 0}, upload=57142]: Segment reached optimal threshold of 14 pieces. Jul 26 23:11:26.324097706 (59h35m57.514396276s) [noise=true , address=104.36.82.46:4499 , piece_id=LMVELJMAGNSYUEXBPBLKELAFY4NHIMKJDLZAQALBQSRLYGUDMG4Q, seg_pos={0 0}, upload=57142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc): protocol: expected piece hash; context canceled; EOF Jul 26 23:11:26.324886152 (59h35m57.515184722s) [noise=true , address=104.36.82.47:4532 , piece_id=QZ45QSEDTLDOX7KFZKJVOMQG2MEMKI2FBFRL66H5GLP6P45TX55A, seg_pos={0 0}, upload=57142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL): protocol: expected piece hash; context canceled; EOF Jul 26 23:11:26.325527404 (59h35m57.515825982s) [noise=true , address=77.90.31.13:4644 , piece_id=A2SL7EGOGFY5ZNBXK37S7GAPYO2ZVGONZBSMXMCADQGDMTPVJXAA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h): protocol: rpc: dial tcp 77.90.31.13:4644: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:26.325566786 (59h35m57.515865373s) [seg_pos={0 0}, upload=57142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 26 23:11:26.325647962 (59h35m57.515946552s) [upload=57142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:11:26.325654211 (59h35m57.515952801s) [upload=57142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:11:26.481578936 (59h35m57.671877516s) [upload=57142]: Done uploading segments... Jul 26 23:11:26.481598726 (59h35m57.671897296s) [upload=57142]: Upload finished. err: <nil> Duration: 1m0.003877713s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X..X...X.XX...X..X.XX......X.X.......X.......X........XX....] duration: 19.922371758s FAIL (2025-07-26 23:11:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.922392821s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................X.XX..] duration: 4.592646816s FAIL (2025-07-26 23:10:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.592686223s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.X..X...X.XX...X..X.XX......X.X.......X.......X........XX...] duration: 59.63592753s FAIL (2025-07-26 23:10:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 59.636028307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [............................X..................X.....XXXXXX.] duration: 23.883356855s FAIL (2025-07-26 23:10:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 23.88338406s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................................X......XXXXXX] duration: 30.102547956s FAIL (2025-07-26 23:10:49 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== EOF Jul 26 23:10:00.246066191 (59h34m31.436364772s) [noise=true , address=104.36.82.4:4803 , piece_id=SQXKU6KLD3ACZ4NWY6BKTNYVSZWLOM3IL3CKDZGBQA2YYEC66SVQ, seg_pos={0 0}, upload=57129]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:10:00.246144497 (59h34m31.436443157s) [seg_pos={0 0}, upload=57129]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:10:00.366578844 (59h34m31.556877416s) [upload=57129]: Done uploading segments... Jul 26 23:10:00.366614776 (59h34m31.556913356s) [upload=57129]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:10:19.317217089 (59h34m50.507515669s) [upload=57141]: Starting upload Jul 26 23:10:19.317277484 (59h34m50.507576067s) [upload=57141]: Done starting upload Jul 26 23:10:19.317342805 (59h34m50.507641375s) [upload=57141]: Uploading segments... Jul 26 23:10:19.317528496 (59h34m50.507827077s) [upload=57141]: Got next segment. Inline: false Jul 26 23:10:19.317570546 (59h34m50.50786918s) [upload=57141]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:10:19.317577417 (59h34m50.507876004s) [upload=57141]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:10:19.510135458 (59h34m50.700434041s) [seg_pos={0 0}, upload=57141]: Begin upload segment... Jul 26 23:10:19.510210196 (59h34m50.700508776s) [seg_pos={0 0}, upload=57141]: Done begin upload segment. Jul 26 23:10:19.510272248 (59h34m50.700570833s) [noise=true , address=104.36.82.4:4803 , piece_id=BFS7SC7QJEARKPW3MIJIKCZD7FWIO56A5TJK4OTCKVEPQMXTPOXA, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.511212408 (59h34m50.701510988s) [noise=true , address=104.36.82.8:4969 , piece_id=25TJDTOTBCVIIM2O3GZC2YYFDY7EQEHI2BQKJDVKEXOPYQX72DRA, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.511733837 (59h34m50.702032419s) [noise=true , address=104.36.82.4:4802 , piece_id=GLDLHWZSZBKAC72TGOLKDHEGSAU6GDCGZTRMW62SSNOD2GGHW2MQ, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.512122933 (59h34m50.702421513s) [noise=true , address=45.91.116.6:4973 , piece_id=DVMJCXB52267NQ743A4J7NKN3Z7YKAF4YCAG2QBGXOEKPGFMX67A, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.512639953 (59h34m50.702938539s) [noise=true , address=104.36.82.16:4295 , piece_id=MPWNPA76ASME7JM3M6DINVCSVUWNQH6NMXNTCJT3H7GYB54TQZOQ, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.513037199 (59h34m50.703335782s) [noise=true , address=104.36.82.14:4221 , piece_id=HGENQ27EM6BSI6MD4YZLLXLPIGWFKM543HUO6A3C65EC7UDLABDQ, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.513409000 (59h34m50.703707582s) [noise=true , address=45.91.116.13:4155 , piece_id=CDBYGUE247JKJAXHMW4T7XN7KP7PJ2PHZJT4YZEKI3277OIHJ2HA, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.513776657 (59h34m50.704075239s) [noise=true , address=45.91.116.5:4298 , piece_id=MSVBJHSELN3MHOUFIC5EKIVEKUR4RVVTEBDYUIGY5264IWMPK7FQ, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.513997356 (59h34m50.704295991s) [noise=true , address=77.90.31.15:4698 , piece_id=NCR5SAPXR2JDPWR6IK2LDVDH4ZVNFGOUPAZGSIMCU5ZGENM4SLWQ, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.514463779 (59h34m50.704762362s) [noise=true , address=104.36.82.15:4258 , piece_id=JIJCVHYIV7H22VFC4XKGTILRLJFFNWIQ6PNBXNNAYN5CGTNXGHIA, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.514849424 (59h34m50.705148013s) [noise=true , address=104.36.82.14:4223 , piece_id=MQWVXOPRBXJZQOJW3GSDYZ7HOEY3OCM77TZQNWJF6ID46QQ6B35Q, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.515222576 (59h34m50.705521167s) [noise=true , address=45.91.116.8:4047 , piece_id=N47UXVNVONPGRBEBHMATN2B5LMB3TV6GDT4YLT4OSTFLYDTLM72A, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.515582999 (59h34m50.705881578s) [noise=true , address=104.36.82.6:4874 , piece_id=ZWIJXARR5KNH5FXTJOND6WOA73SF66FDF2DWHOLBUNTGKGWSMSXQ, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.515911687 (59h34m50.706210266s) [noise=true , address=77.90.31.14:4662 , piece_id=QU4AV4Y7OHGEGRL34CVFCV4ELB234VRGQWLAJD7O4265PLRA3YPQ, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.516224830 (59h34m50.706523412s) [noise=true , address=104.36.82.2:4729 , piece_id=MMMVWC3Q7WS6METIBMRJF2UP4DNXQJ6C3OGUERGYDUX567UM3YGA, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:19.516702874 (59h34m50.707001463s) [noise=true , address=104.36.82.11:4114 , piece_id=IJQKNT77YNUJUYJDPTDBIF6O5R3M5PSE3FDF4NE4CBMGISMUW7JQ, seg_pos={0 0}, upload=57141]: Uploading piece... Jul 26 23:10:20.128776168 (59h34m51.319074741s) [upload=57141]: Next returned nil segment Jul 26 23:10:20.128794373 (59h34m51.319092954s) [upload=57141]: Waiting for error group managing segments... Jul 26 23:10:20.266707671 (59h34m51.457006243s) [noise=true , address=45.91.116.13:4155 , piece_id=CDBYGUE247JKJAXHMW4T7XN7KP7PJ2PHZJT4YZEKI3277OIHJ2HA, seg_pos={0 0}, upload=57141]: Done uploading piece. err: <nil> Jul 26 23:10:20.274765194 (59h34m51.46506376s) [noise=true , address=45.91.116.8:4047 , piece_id=N47UXVNVONPGRBEBHMATN2B5LMB3TV6GDT4YLT4OSTFLYDTLM72A, seg_pos={0 0}, upload=57141]: Done uploading piece. err: <nil> Jul 26 23:10:20.276647687 (59h34m51.466946256s) [noise=true , address=45.91.116.6:4973 , piece_id=DVMJCXB52267NQ743A4J7NKN3Z7YKAF4YCAG2QBGXOEKPGFMX67A, seg_pos={0 0}, upload=57141]: Done uploading piece. err: <nil> Jul 26 23:10:20.277078761 (59h34m51.467377352s) [noise=true , address=45.91.116.5:4298 , piece_id=MSVBJHSELN3MHOUFIC5EKIVEKUR4RVVTEBDYUIGY5264IWMPK7FQ, seg_pos={0 0}, upload=57141]: Done uploading piece. err: <nil> Jul 26 23:10:20.286292268 (59h34m51.476590855s) [noise=true , address=77.90.31.14:4662 , piece_id=QU4AV4Y7OHGEGRL34CVFCV4ELB234VRGQWLAJD7O4265PLRA3YPQ, seg_pos={0 0}, upload=57141]: Done uploading piece. err: <nil> Jul 26 23:10:21.534337028 (59h34m52.724635609s) [noise=true , address=77.90.31.15:4698 , piece_id=NCR5SAPXR2JDPWR6IK2LDVDH4ZVNFGOUPAZGSIMCU5ZGENM4SLWQ, seg_pos={0 0}, upload=57141]: Done uploading piece. err: <nil> Jul 26 23:10:34.156525339 (59h35m5.346823904s) [noise=true , address=104.36.82.16:4295 , piece_id=MPWNPA76ASME7JM3M6DINVCSVUWNQH6NMXNTCJT3H7GYB54TQZOQ, seg_pos={0 0}, upload=57141]: Done uploading piece. err: <nil> Jul 26 23:10:39.469282117 (59h35m10.659580698s) [noise=true , address=104.36.82.6:4874 , piece_id=ZWIJXARR5KNH5FXTJOND6WOA73SF66FDF2DWHOLBUNTGKGWSMSXQ, seg_pos={0 0}, upload=57141]: Done uploading piece. err: <nil> Jul 26 23:10:42.761973185 (59h35m13.952271764s) [noise=true , address=104.36.82.2:4729 , piece_id=MMMVWC3Q7WS6METIBMRJF2UP4DNXQJ6C3OGUERGYDUX567UM3YGA, seg_pos={0 0}, upload=57141]: Done uploading piece. err: <nil> Jul 26 23:10:42.789304236 (59h35m13.979602801s) [noise=true , address=104.36.82.15:4258 , piece_id=JIJCVHYIV7H22VFC4XKGTILRLJFFNWIQ6PNBXNNAYN5CGTNXGHIA, seg_pos={0 0}, upload=57141]: Done uploading piece. err: <nil> Jul 26 23:10:44.657045430 (59h35m15.847344018s) [noise=true , address=104.36.82.4:4803 , piece_id=BFS7SC7QJEARKPW3MIJIKCZD7FWIO56A5TJK4OTCKVEPQMXTPOXA, seg_pos={0 0}, upload=57141]: Done uploading piece. err: <nil> Jul 26 23:10:44.856377454 (59h35m16.046676038s) [noise=true , address=104.36.82.4:4802 , piece_id=GLDLHWZSZBKAC72TGOLKDHEGSAU6GDCGZTRMW62SSNOD2GGHW2MQ, seg_pos={0 0}, upload=57141]: Done uploading piece. err: <nil> Jul 26 23:10:48.868010949 (59h35m20.058309532s) [noise=true , address=104.36.82.14:4221 , piece_id=HGENQ27EM6BSI6MD4YZLLXLPIGWFKM543HUO6A3C65EC7UDLABDQ, seg_pos={0 0}, upload=57141]: Done uploading piece. err: <nil> Jul 26 23:10:48.868049178 (59h35m20.058347759s) [seg_pos={0 0}, upload=57141]: Segment reached optimal threshold of 13 pieces. Jul 26 23:10:48.868365898 (59h35m20.058664476s) [noise=true , address=104.36.82.11:4114 , piece_id=IJQKNT77YNUJUYJDPTDBIF6O5R3M5PSE3FDF4NE4CBMGISMUW7JQ, seg_pos={0 0}, upload=57141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:35604->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:35604->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35604->104.36.82.11:4114: use of closed network connection Jul 26 23:10:48.868690218 (59h35m20.058988799s) [noise=true , address=104.36.82.8:4969 , piece_id=25TJDTOTBCVIIM2O3GZC2YYFDY7EQEHI2BQKJDVKEXOPYQX72DRA, seg_pos={0 0}, upload=57141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 26 23:10:48.868933897 (59h35m20.059232471s) [noise=true , address=104.36.82.14:4223 , piece_id=MQWVXOPRBXJZQOJW3GSDYZ7HOEY3OCM77TZQNWJF6ID46QQ6B35Q, seg_pos={0 0}, upload=57141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: expected piece hash; context canceled; EOF Jul 26 23:10:48.868957088 (59h35m20.059255677s) [seg_pos={0 0}, upload=57141]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 23:10:48.869081778 (59h35m20.059380367s) [upload=57141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:10:48.869088819 (59h35m20.059387406s) [upload=57141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:10:49.135006620 (59h35m20.325305195s) [upload=57141]: Done uploading segments... Jul 26 23:10:49.135055934 (59h35m20.325354514s) [upload=57141]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.102573936s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............................X..................X.....XXXXXX] duration: 30.11726874s FAIL (2025-07-26 23:10:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.117292084s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................X......XXXXX] duration: 30.124382833s FAIL (2025-07-26 23:10:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== least 13 but got 10) Jul 26 23:09:30.523848465 (59h34m1.714147053s) [upload=57129]: Starting upload Jul 26 23:09:30.523970673 (59h34m1.714269257s) [upload=57129]: Done starting upload Jul 26 23:09:30.524123664 (59h34m1.714422249s) [upload=57129]: Uploading segments... Jul 26 23:09:30.524558317 (59h34m1.714856905s) [upload=57129]: Got next segment. Inline: false Jul 26 23:09:30.524640130 (59h34m1.71493872s) [upload=57129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:09:30.524646668 (59h34m1.714945258s) [upload=57129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:09:30.640642238 (59h34m1.830940845s) [seg_pos={0 0}, upload=57129]: Begin upload segment... Jul 26 23:09:30.640715911 (59h34m1.831014501s) [seg_pos={0 0}, upload=57129]: Done begin upload segment. Jul 26 23:09:30.640747999 (59h34m1.831046588s) [noise=true , address=45.91.116.10:4081 , piece_id=JJ32JVJKFIJLFQAZ6VUJRJC3IJZWDXSUKUF3PMBTJMBQVZIZ3NYA, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.641066280 (59h34m1.831364848s) [noise=true , address=45.91.116.4:4940 , piece_id=MOGNCEHISGMNNDSR4QT4ZLRMEZUJEGVLE3O6YOBQC4JCW63C6GTA, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.641887467 (59h34m1.832186098s) [noise=true , address=45.91.116.4:4937 , piece_id=UW6URD5G5IS6AW4WLGANEBKJMV5OGGI22XXPX7QMMJRKMKHMWJQA, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.642719294 (59h34m1.833017861s) [noise=true , address=77.90.31.12:4589 , piece_id=UD4M4XI4WRWBKYVJKPHWWS345DPXRCBYORYM5XW5HPHOB37GYWHQ, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.642843226 (59h34m1.833141793s) [noise=true , address=104.36.82.2:4729 , piece_id=BLFSHZLS2KC6POFOE274UUA2ROWHYXHDHSFXM3J5G5SDIIDC4YWA, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.643163184 (59h34m1.83346175s) [noise=true , address=104.36.82.12:4151 , piece_id=NNFV3Q2SHBFYWEPYIQNBLLGUWPDLYA7QE2BEXE4OU76NHNVXEKCA, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.643398210 (59h34m1.833696783s) [noise=true , address=104.36.82.9:4007 , piece_id=USIBRSCGTLQ6PV3ZBW57NA34LMDKHFK4VC73I5OUOYWWZKNXKDEQ, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.643709411 (59h34m1.834007984s) [noise=true , address=104.36.82.11:4114 , piece_id=2PVFKUU4G25IKU7F7F6B3ZKR5SPTGFZRCCD5QXYABJEAPKNJK56A, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.644030835 (59h34m1.834329407s) [noise=true , address=104.36.82.4:4803 , piece_id=SQXKU6KLD3ACZ4NWY6BKTNYVSZWLOM3IL3CKDZGBQA2YYEC66SVQ, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.644166699 (59h34m1.834465273s) [noise=true , address=77.90.31.11:4553 , piece_id=WPDLXQ5MGXSNCAXU5YAZH7XMAJ45DIKPKTZ6DQYO2NGA7G7YRFHA, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.644716659 (59h34m1.835015226s) [noise=true , address=104.36.82.3:4765 , piece_id=OJNGV47AK7KEKX3SBNSWTZI4AIDOI2C6QP3RGA4NZYJT2XAOVTXA, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.644961106 (59h34m1.83525969s) [noise=true , address=104.36.82.4:4804 , piece_id=VOKIZLTU776C6WKWZQFGZVNWQN2X5SZ6QXGRHKRHGEZPYTWI5V6Q, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.645221794 (59h34m1.835520367s) [noise=true , address=45.91.116.6:4975 , piece_id=CZNWAJUSE7ESQR5NSMJMDFXPL4NTCQNK33J5LIT3V3QIRGAAWKPQ, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.645526385 (59h34m1.835824958s) [noise=true , address=45.91.116.11:4118 , piece_id=EEQZMJADU66RG3DFFTVBRDKSX4E2OFG7TS2K624IF6TDWVFEWEEA, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.645692609 (59h34m1.835991185s) [noise=true , address=45.91.116.5:4299 , piece_id=JHYSJRWFENQRV3R5HOWGGURHVGVSPHSLE4LFZT3B4ACZXCM2MC2Q, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:30.645969827 (59h34m1.836268394s) [noise=true , address=104.36.82.15:4258 , piece_id=BY4E2WENJGPI44OXWBGCKWVLCRKBCTUQN4YICKOBEGADJ6TZMOAQ, seg_pos={0 0}, upload=57129]: Uploading piece... Jul 26 23:09:31.248891752 (59h34m2.439190331s) [upload=57129]: Next returned nil segment Jul 26 23:09:31.248909273 (59h34m2.439207858s) [upload=57129]: Waiting for error group managing segments... Jul 26 23:09:31.387233096 (59h34m2.577531677s) [noise=true , address=45.91.116.6:4975 , piece_id=CZNWAJUSE7ESQR5NSMJMDFXPL4NTCQNK33J5LIT3V3QIRGAAWKPQ, seg_pos={0 0}, upload=57129]: Done uploading piece. err: <nil> Jul 26 23:09:31.389180315 (59h34m2.579478895s) [noise=true , address=45.91.116.10:4081 , piece_id=JJ32JVJKFIJLFQAZ6VUJRJC3IJZWDXSUKUF3PMBTJMBQVZIZ3NYA, seg_pos={0 0}, upload=57129]: Done uploading piece. err: <nil> Jul 26 23:09:31.390640490 (59h34m2.580939072s) [noise=true , address=45.91.116.11:4118 , piece_id=EEQZMJADU66RG3DFFTVBRDKSX4E2OFG7TS2K624IF6TDWVFEWEEA, seg_pos={0 0}, upload=57129]: Done uploading piece. err: <nil> Jul 26 23:09:31.397244191 (59h34m2.587542769s) [noise=true , address=45.91.116.4:4937 , piece_id=UW6URD5G5IS6AW4WLGANEBKJMV5OGGI22XXPX7QMMJRKMKHMWJQA, seg_pos={0 0}, upload=57129]: Done uploading piece. err: <nil> Jul 26 23:09:31.405800212 (59h34m2.5960988s) [noise=true , address=45.91.116.5:4299 , piece_id=JHYSJRWFENQRV3R5HOWGGURHVGVSPHSLE4LFZT3B4ACZXCM2MC2Q, seg_pos={0 0}, upload=57129]: Done uploading piece. err: <nil> Jul 26 23:09:31.409400883 (59h34m2.599699454s) [noise=true , address=77.90.31.11:4553 , piece_id=WPDLXQ5MGXSNCAXU5YAZH7XMAJ45DIKPKTZ6DQYO2NGA7G7YRFHA, seg_pos={0 0}, upload=57129]: Done uploading piece. err: <nil> Jul 26 23:09:31.420030552 (59h34m2.610329122s) [noise=true , address=45.91.116.4:4940 , piece_id=MOGNCEHISGMNNDSR4QT4ZLRMEZUJEGVLE3O6YOBQC4JCW63C6GTA, seg_pos={0 0}, upload=57129]: Done uploading piece. err: <nil> Jul 26 23:10:00.244820803 (59h34m31.435119383s) [noise=true , address=104.36.82.9:4007 , piece_id=USIBRSCGTLQ6PV3ZBW57NA34LMDKHFK4VC73I5OUOYWWZKNXKDEQ, seg_pos={0 0}, upload=57129]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:10:00.245314790 (59h34m31.4356134s) [noise=true , address=104.36.82.12:4151 , piece_id=NNFV3Q2SHBFYWEPYIQNBLLGUWPDLYA7QE2BEXE4OU76NHNVXEKCA, seg_pos={0 0}, upload=57129]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:10:00.245433496 (59h34m31.435732077s) [noise=true , address=104.36.82.4:4804 , piece_id=VOKIZLTU776C6WKWZQFGZVNWQN2X5SZ6QXGRHKRHGEZPYTWI5V6Q, seg_pos={0 0}, upload=57129]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:10:00.245709140 (59h34m31.436007721s) [noise=true , address=77.90.31.12:4589 , piece_id=UD4M4XI4WRWBKYVJKPHWWS345DPXRCBYORYM5XW5HPHOB37GYWHQ, seg_pos={0 0}, upload=57129]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:00.245827460 (59h34m31.436126047s) [noise=true , address=104.36.82.3:4765 , piece_id=OJNGV47AK7KEKX3SBNSWTZI4AIDOI2C6QP3RGA4NZYJT2XAOVTXA, seg_pos={0 0}, upload=57129]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 139.59.144.22:60624->104.36.82.3:4765: use of closed network connection; write tcp 139.59.144.22:60624->104.36.82.3:4765: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60624->104.36.82.3:4765: use of closed network connection Jul 26 23:10:00.245870633 (59h34m31.436169216s) [noise=true , address=104.36.82.15:4258 , piece_id=BY4E2WENJGPI44OXWBGCKWVLCRKBCTUQN4YICKOBEGADJ6TZMOAQ, seg_pos={0 0}, upload=57129]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:10:00.246008759 (59h34m31.436307342s) [noise=true , address=104.36.82.11:4114 , piece_id=2PVFKUU4G25IKU7F7F6B3ZKR5SPTGFZRCCD5QXYABJEAPKNJK56A, seg_pos={0 0}, upload=57129]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: read tcp 139.59.144.22:55738->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:00.246041566 (59h34m31.436340146s) [noise=true , address=104.36.82.2:4729 , piece_id=BLFSHZLS2KC6POFOE274UUA2ROWHYXHDHSFXM3J5G5SDIIDC4YWA, seg_pos={0 0}, upload=57129]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:10:00.246066191 (59h34m31.436364772s) [noise=true , address=104.36.82.4:4803 , piece_id=SQXKU6KLD3ACZ4NWY6BKTNYVSZWLOM3IL3CKDZGBQA2YYEC66SVQ, seg_pos={0 0}, upload=57129]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:10:00.246144497 (59h34m31.436443157s) [seg_pos={0 0}, upload=57129]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:10:00.366578844 (59h34m31.556877416s) [upload=57129]: Done uploading segments... Jul 26 23:10:00.366614776 (59h34m31.556913356s) [upload=57129]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.124420543s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [..XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.XX..] duration: 22.405237905s FAIL (2025-07-26 23:09:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== GIMYXFQQUJXF6MER4IFK5MFM4FVWGQEID2IMA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.494281893 (59h32m25.684580477s) [noise=true , address=104.36.82.46:4488 , piece_id=BXZDZKONUW7KZHSDZG2W36WZZAQGRD3L7WMAHC4ATHVNYKXYMSOQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.494647291 (59h32m25.684945857s) [noise=true , address=104.36.82.55:4228 , piece_id=H5R3RO2QR2MY442KURXZJMV2VHLGUNM3VGPDBCNV2AFPC2TL7H7A, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.494680050 (59h32m25.684978632s) [noise=true , address=104.36.82.10:4092 , piece_id=BI4I6KA7FMXLZRYCOMUZKLOOCMIG5BG7P7WFOB55AF24M2HVXBJA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.494963704 (59h32m25.685262289s) [noise=true , address=104.36.82.41:4792 , piece_id=UZXN3XBIZG3RI7UML7DLGT42Q232VY35KVX5XDSG2E32PSGSP7UA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.495204799 (59h32m25.685503381s) [noise=true , address=104.36.82.52:4102 , piece_id=DJCRQB7TLRCRYGQTINCXKH4OVEDZUGL4WZQV6UHIOTWHPZT34TOQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.495450502 (59h32m25.685749081s) [noise=true , address=104.36.82.51:4082 , piece_id=JKZ66BI4FIUHOYCHE2CNB5PNB6EMM4PKM3NLVGB6MQC6CS4YPH5Q, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.495734441 (59h32m25.686033026s) [noise=true , address=104.36.82.47:4544 , piece_id=BOGWO4ZBYXNPZAWDSVSURLCGZQH6CCKWOFEHVKPICRZE6XTDQL7Q, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.496014873 (59h32m25.68631347s) [noise=true , address=104.36.82.42:4819 , piece_id=5S7YOGIGMMNUYFXDKGSHDHPQKVEUK5C4TTFB5B36ZGJ5EGJ4F4UA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.496408322 (59h32m25.686706901s) [noise=true , address=104.36.82.56:4250 , piece_id=GPOOLCCNXPBX3ANVS5TYMKIVR6FCQX2OIY2JJN4FW7VDH66ZRUIA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.496915355 (59h32m25.687213936s) [noise=true , address=104.36.82.10:4096 , piece_id=6YVLJBU4LUU7N5CW36RG2YJRSCVNWAZ36OCGDLR733PJJGIBD3YA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.497019891 (59h32m25.687318482s) [noise=true , address=79.127.219.38:4006 , piece_id=IX24VYV3A2VI4P7V7VFPSU3IB26DGK6CB6ZR5BNRIJEECA5SFLGA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.497410087 (59h32m25.687708667s) [noise=true , address=104.36.82.54:4161 , piece_id=Q4IJZW2SNBY5EM5YCNDMOROGUDT3H5IV3QY3YVYC7PD2RRYZ3OHA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.497781243 (59h32m25.688079826s) [noise=true , address=45.91.116.14:4194 , piece_id=Y3LFPXJEPRSURNBBTI2WKXXDMB5U4PFLKFT5EEMNS5V3R3CO63FQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.498058987 (59h32m25.688357565s) [noise=true , address=104.36.82.36:4624 , piece_id=IMQXVWLVQP4PLRZAZIAFJXQS5CZQOIWSV7KJYPCUGUZ2Y4M2QQHA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.498389855 (59h32m25.688688433s) [noise=true , address=77.90.31.12:4611 , piece_id=V56NZRFEZH7SN6A2E3S5JNA3B2I5SGWF5YX4YRMSJTHXR5HEON6A, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.498668718 (59h32m25.688967296s) [noise=true , address=104.36.82.21:4482 , piece_id=UCBALDOS3OIL66RR4VKBRR6EVITG6ZF4YYO2GRJRX3L4AXUKAFXQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.498761422 (59h32m25.689060002s) [noise=true , address=79.127.201.215:4356 , piece_id=KAOMOYSUS2YH45BH4TVLK62A5AVFLIEZPQEPRHOSK5LVAXZB2YHQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:55.074701029 (59h32m26.264999615s) [upload=57098]: Next returned nil segment Jul 26 23:07:55.074724792 (59h32m26.26502339s) [upload=57098]: Waiting for error group managing segments... Jul 26 23:07:55.216394130 (59h32m26.4066927s) [noise=true , address=45.91.116.14:4194 , piece_id=Y3LFPXJEPRSURNBBTI2WKXXDMB5U4PFLKFT5EEMNS5V3R3CO63FQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.231153209 (59h32m26.421451775s) [noise=true , address=104.36.82.52:4102 , piece_id=DJCRQB7TLRCRYGQTINCXKH4OVEDZUGL4WZQV6UHIOTWHPZT34TOQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.235588310 (59h32m26.425886883s) [noise=true , address=104.36.82.54:4161 , piece_id=Q4IJZW2SNBY5EM5YCNDMOROGUDT3H5IV3QY3YVYC7PD2RRYZ3OHA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.236945932 (59h32m26.427244504s) [noise=true , address=104.36.82.55:4228 , piece_id=H5R3RO2QR2MY442KURXZJMV2VHLGUNM3VGPDBCNV2AFPC2TL7H7A, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.252295061 (59h32m26.44259363s) [noise=true , address=104.36.82.56:4250 , piece_id=GPOOLCCNXPBX3ANVS5TYMKIVR6FCQX2OIY2JJN4FW7VDH66ZRUIA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.314244807 (59h32m26.504543373s) [noise=true , address=79.127.219.38:4006 , piece_id=IX24VYV3A2VI4P7V7VFPSU3IB26DGK6CB6ZR5BNRIJEECA5SFLGA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:58.098342640 (59h32m29.288641211s) [noise=true , address=104.36.82.47:4544 , piece_id=BOGWO4ZBYXNPZAWDSVSURLCGZQH6CCKWOFEHVKPICRZE6XTDQL7Q, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:01.042437350 (59h32m32.232735935s) [noise=true , address=104.36.82.46:4488 , piece_id=BXZDZKONUW7KZHSDZG2W36WZZAQGRD3L7WMAHC4ATHVNYKXYMSOQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:04.364883316 (59h32m35.555181895s) [noise=true , address=79.127.201.215:4356 , piece_id=KAOMOYSUS2YH45BH4TVLK62A5AVFLIEZPQEPRHOSK5LVAXZB2YHQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:08.994662985 (59h32m40.184961558s) [noise=true , address=79.127.201.215:4355 , piece_id=725MUVVECDVEHOHGIMYXFQQUJXF6MER4IFK5MFM4FVWGQEID2IMA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:44.699576901 (59h33m15.889875474s) [noise=true , address=104.36.82.10:4092 , piece_id=BI4I6KA7FMXLZRYCOMUZKLOOCMIG5BG7P7WFOB55AF24M2HVXBJA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:53.938251203 (59h33m25.128549772s) [noise=true , address=104.36.82.51:4082 , piece_id=JKZ66BI4FIUHOYCHE2CNB5PNB6EMM4PKM3NLVGB6MQC6CS4YPH5Q, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:1m5NjTfYFQoyaftkARd75HMFZDW1jpnVj7H8LaUZWZcZYozHru, address:104.36.82.51:4082): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:08:53.938467746 (59h33m25.128766316s) [noise=true , address=104.36.82.10:4096 , piece_id=6YVLJBU4LUU7N5CW36RG2YJRSCVNWAZ36OCGDLR733PJJGIBD3YA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA, address:104.36.82.10:4096): protocol: write tcp 139.59.144.22:37300->104.36.82.10:4096: use of closed network connection; write tcp 139.59.144.22:37300->104.36.82.10:4096: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37300->104.36.82.10:4096: use of closed network connection Jul 26 23:08:53.938546369 (59h33m25.128844952s) [noise=true , address=104.36.82.21:4482 , piece_id=UCBALDOS3OIL66RR4VKBRR6EVITG6ZF4YYO2GRJRX3L4AXUKAFXQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:12B2s428CMVK7mdLV84VgWFcdRQaVQxcA7YuqqeqEM8FAefEGt7, address:104.36.82.21:4482): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:08:53.938573528 (59h33m25.128872098s) [noise=true , address=104.36.82.36:4624 , piece_id=IMQXVWLVQP4PLRZAZIAFJXQS5CZQOIWSV7KJYPCUGUZ2Y4M2QQHA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1, address:104.36.82.36:4624): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:08:53.938624398 (59h33m25.128922967s) [noise=true , address=77.90.31.12:4611 , piece_id=V56NZRFEZH7SN6A2E3S5JNA3B2I5SGWF5YX4YRMSJTHXR5HEON6A, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:1r7qdqiUuUqsKnTXdC3S1Ek84rsCPtjPoXbPugVHWBzJezv4W2, address:77.90.31.12:4611): protocol: rpc: dial tcp 77.90.31.12:4611: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:53.938673213 (59h33m25.128971786s) [noise=true , address=104.36.82.41:4792 , piece_id=UZXN3XBIZG3RI7UML7DLGT42Q232VY35KVX5XDSG2E32PSGSP7UA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k, address:104.36.82.41:4792): protocol: write tcp 139.59.144.22:45786->104.36.82.41:4792: use of closed network connection; write tcp 139.59.144.22:45786->104.36.82.41:4792: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45786->104.36.82.41:4792: use of closed network connection Jul 26 23:08:53.938718991 (59h33m25.129017561s) [noise=true , address=104.36.82.42:4819 , piece_id=5S7YOGIGMMNUYFXDKGSHDHPQKVEUK5C4TTFB5B36ZGJ5EGJ4F4UA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:12MZhrDu1fjMN7XFoR2h1q3JPLHJYYURGQwQgcgLuucSrGShjoS, address:104.36.82.42:4819): protocol: write tcp 139.59.144.22:36980->104.36.82.42:4819: use of closed network connection; write tcp 139.59.144.22:36980->104.36.82.42:4819: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36980->104.36.82.42:4819: use of closed network connection Jul 26 23:08:53.938738074 (59h33m25.129036655s) [seg_pos={0 0}, upload=57098]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:08:54.045356315 (59h33m25.235654887s) [upload=57098]: Done uploading segments... Jul 26 23:08:54.045383269 (59h33m25.235681848s) [upload=57098]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 22.405267794s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................X.XX.] duration: 2.885054813s FAIL (2025-07-26 23:09:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.885095576s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..X..X...X.XX...X..X.XX......X.X.......X.......X........XX..] duration: 22.855958115s FAIL (2025-07-26 23:09:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.855990818s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................X.XX] duration: 30.001558351s FAIL (2025-07-26 23:09:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001589694s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [...XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.XX.] duration: 28.879355219s FAIL (2025-07-26 23:09:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== GIMYXFQQUJXF6MER4IFK5MFM4FVWGQEID2IMA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.494281893 (59h32m25.684580477s) [noise=true , address=104.36.82.46:4488 , piece_id=BXZDZKONUW7KZHSDZG2W36WZZAQGRD3L7WMAHC4ATHVNYKXYMSOQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.494647291 (59h32m25.684945857s) [noise=true , address=104.36.82.55:4228 , piece_id=H5R3RO2QR2MY442KURXZJMV2VHLGUNM3VGPDBCNV2AFPC2TL7H7A, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.494680050 (59h32m25.684978632s) [noise=true , address=104.36.82.10:4092 , piece_id=BI4I6KA7FMXLZRYCOMUZKLOOCMIG5BG7P7WFOB55AF24M2HVXBJA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.494963704 (59h32m25.685262289s) [noise=true , address=104.36.82.41:4792 , piece_id=UZXN3XBIZG3RI7UML7DLGT42Q232VY35KVX5XDSG2E32PSGSP7UA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.495204799 (59h32m25.685503381s) [noise=true , address=104.36.82.52:4102 , piece_id=DJCRQB7TLRCRYGQTINCXKH4OVEDZUGL4WZQV6UHIOTWHPZT34TOQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.495450502 (59h32m25.685749081s) [noise=true , address=104.36.82.51:4082 , piece_id=JKZ66BI4FIUHOYCHE2CNB5PNB6EMM4PKM3NLVGB6MQC6CS4YPH5Q, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.495734441 (59h32m25.686033026s) [noise=true , address=104.36.82.47:4544 , piece_id=BOGWO4ZBYXNPZAWDSVSURLCGZQH6CCKWOFEHVKPICRZE6XTDQL7Q, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.496014873 (59h32m25.68631347s) [noise=true , address=104.36.82.42:4819 , piece_id=5S7YOGIGMMNUYFXDKGSHDHPQKVEUK5C4TTFB5B36ZGJ5EGJ4F4UA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.496408322 (59h32m25.686706901s) [noise=true , address=104.36.82.56:4250 , piece_id=GPOOLCCNXPBX3ANVS5TYMKIVR6FCQX2OIY2JJN4FW7VDH66ZRUIA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.496915355 (59h32m25.687213936s) [noise=true , address=104.36.82.10:4096 , piece_id=6YVLJBU4LUU7N5CW36RG2YJRSCVNWAZ36OCGDLR733PJJGIBD3YA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.497019891 (59h32m25.687318482s) [noise=true , address=79.127.219.38:4006 , piece_id=IX24VYV3A2VI4P7V7VFPSU3IB26DGK6CB6ZR5BNRIJEECA5SFLGA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.497410087 (59h32m25.687708667s) [noise=true , address=104.36.82.54:4161 , piece_id=Q4IJZW2SNBY5EM5YCNDMOROGUDT3H5IV3QY3YVYC7PD2RRYZ3OHA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.497781243 (59h32m25.688079826s) [noise=true , address=45.91.116.14:4194 , piece_id=Y3LFPXJEPRSURNBBTI2WKXXDMB5U4PFLKFT5EEMNS5V3R3CO63FQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.498058987 (59h32m25.688357565s) [noise=true , address=104.36.82.36:4624 , piece_id=IMQXVWLVQP4PLRZAZIAFJXQS5CZQOIWSV7KJYPCUGUZ2Y4M2QQHA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.498389855 (59h32m25.688688433s) [noise=true , address=77.90.31.12:4611 , piece_id=V56NZRFEZH7SN6A2E3S5JNA3B2I5SGWF5YX4YRMSJTHXR5HEON6A, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.498668718 (59h32m25.688967296s) [noise=true , address=104.36.82.21:4482 , piece_id=UCBALDOS3OIL66RR4VKBRR6EVITG6ZF4YYO2GRJRX3L4AXUKAFXQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.498761422 (59h32m25.689060002s) [noise=true , address=79.127.201.215:4356 , piece_id=KAOMOYSUS2YH45BH4TVLK62A5AVFLIEZPQEPRHOSK5LVAXZB2YHQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:55.074701029 (59h32m26.264999615s) [upload=57098]: Next returned nil segment Jul 26 23:07:55.074724792 (59h32m26.26502339s) [upload=57098]: Waiting for error group managing segments... Jul 26 23:07:55.216394130 (59h32m26.4066927s) [noise=true , address=45.91.116.14:4194 , piece_id=Y3LFPXJEPRSURNBBTI2WKXXDMB5U4PFLKFT5EEMNS5V3R3CO63FQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.231153209 (59h32m26.421451775s) [noise=true , address=104.36.82.52:4102 , piece_id=DJCRQB7TLRCRYGQTINCXKH4OVEDZUGL4WZQV6UHIOTWHPZT34TOQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.235588310 (59h32m26.425886883s) [noise=true , address=104.36.82.54:4161 , piece_id=Q4IJZW2SNBY5EM5YCNDMOROGUDT3H5IV3QY3YVYC7PD2RRYZ3OHA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.236945932 (59h32m26.427244504s) [noise=true , address=104.36.82.55:4228 , piece_id=H5R3RO2QR2MY442KURXZJMV2VHLGUNM3VGPDBCNV2AFPC2TL7H7A, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.252295061 (59h32m26.44259363s) [noise=true , address=104.36.82.56:4250 , piece_id=GPOOLCCNXPBX3ANVS5TYMKIVR6FCQX2OIY2JJN4FW7VDH66ZRUIA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.314244807 (59h32m26.504543373s) [noise=true , address=79.127.219.38:4006 , piece_id=IX24VYV3A2VI4P7V7VFPSU3IB26DGK6CB6ZR5BNRIJEECA5SFLGA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:58.098342640 (59h32m29.288641211s) [noise=true , address=104.36.82.47:4544 , piece_id=BOGWO4ZBYXNPZAWDSVSURLCGZQH6CCKWOFEHVKPICRZE6XTDQL7Q, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:01.042437350 (59h32m32.232735935s) [noise=true , address=104.36.82.46:4488 , piece_id=BXZDZKONUW7KZHSDZG2W36WZZAQGRD3L7WMAHC4ATHVNYKXYMSOQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:04.364883316 (59h32m35.555181895s) [noise=true , address=79.127.201.215:4356 , piece_id=KAOMOYSUS2YH45BH4TVLK62A5AVFLIEZPQEPRHOSK5LVAXZB2YHQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:08.994662985 (59h32m40.184961558s) [noise=true , address=79.127.201.215:4355 , piece_id=725MUVVECDVEHOHGIMYXFQQUJXF6MER4IFK5MFM4FVWGQEID2IMA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:44.699576901 (59h33m15.889875474s) [noise=true , address=104.36.82.10:4092 , piece_id=BI4I6KA7FMXLZRYCOMUZKLOOCMIG5BG7P7WFOB55AF24M2HVXBJA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:53.938251203 (59h33m25.128549772s) [noise=true , address=104.36.82.51:4082 , piece_id=JKZ66BI4FIUHOYCHE2CNB5PNB6EMM4PKM3NLVGB6MQC6CS4YPH5Q, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:1m5NjTfYFQoyaftkARd75HMFZDW1jpnVj7H8LaUZWZcZYozHru, address:104.36.82.51:4082): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:08:53.938467746 (59h33m25.128766316s) [noise=true , address=104.36.82.10:4096 , piece_id=6YVLJBU4LUU7N5CW36RG2YJRSCVNWAZ36OCGDLR733PJJGIBD3YA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA, address:104.36.82.10:4096): protocol: write tcp 139.59.144.22:37300->104.36.82.10:4096: use of closed network connection; write tcp 139.59.144.22:37300->104.36.82.10:4096: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37300->104.36.82.10:4096: use of closed network connection Jul 26 23:08:53.938546369 (59h33m25.128844952s) [noise=true , address=104.36.82.21:4482 , piece_id=UCBALDOS3OIL66RR4VKBRR6EVITG6ZF4YYO2GRJRX3L4AXUKAFXQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:12B2s428CMVK7mdLV84VgWFcdRQaVQxcA7YuqqeqEM8FAefEGt7, address:104.36.82.21:4482): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:08:53.938573528 (59h33m25.128872098s) [noise=true , address=104.36.82.36:4624 , piece_id=IMQXVWLVQP4PLRZAZIAFJXQS5CZQOIWSV7KJYPCUGUZ2Y4M2QQHA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1, address:104.36.82.36:4624): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:08:53.938624398 (59h33m25.128922967s) [noise=true , address=77.90.31.12:4611 , piece_id=V56NZRFEZH7SN6A2E3S5JNA3B2I5SGWF5YX4YRMSJTHXR5HEON6A, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:1r7qdqiUuUqsKnTXdC3S1Ek84rsCPtjPoXbPugVHWBzJezv4W2, address:77.90.31.12:4611): protocol: rpc: dial tcp 77.90.31.12:4611: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:53.938673213 (59h33m25.128971786s) [noise=true , address=104.36.82.41:4792 , piece_id=UZXN3XBIZG3RI7UML7DLGT42Q232VY35KVX5XDSG2E32PSGSP7UA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k, address:104.36.82.41:4792): protocol: write tcp 139.59.144.22:45786->104.36.82.41:4792: use of closed network connection; write tcp 139.59.144.22:45786->104.36.82.41:4792: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45786->104.36.82.41:4792: use of closed network connection Jul 26 23:08:53.938718991 (59h33m25.129017561s) [noise=true , address=104.36.82.42:4819 , piece_id=5S7YOGIGMMNUYFXDKGSHDHPQKVEUK5C4TTFB5B36ZGJ5EGJ4F4UA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:12MZhrDu1fjMN7XFoR2h1q3JPLHJYYURGQwQgcgLuucSrGShjoS, address:104.36.82.42:4819): protocol: write tcp 139.59.144.22:36980->104.36.82.42:4819: use of closed network connection; write tcp 139.59.144.22:36980->104.36.82.42:4819: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36980->104.36.82.42:4819: use of closed network connection Jul 26 23:08:53.938738074 (59h33m25.129036655s) [seg_pos={0 0}, upload=57098]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:08:54.045356315 (59h33m25.235654887s) [upload=57098]: Done uploading segments... Jul 26 23:08:54.045383269 (59h33m25.235681848s) [upload=57098]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 28.879382294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.............................X..................X.....XXXXX] duration: 30.133382378s FAIL (2025-07-26 23:09:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.133455148s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................X......XXXX] duration: 30.111330038s FAIL (2025-07-26 23:09:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== d=57112]: Starting upload Jul 26 23:08:47.043169347 (59h33m18.233467929s) [upload=57112]: Done starting upload Jul 26 23:08:47.043508115 (59h33m18.233806684s) [upload=57112]: Uploading segments... Jul 26 23:08:47.043569245 (59h33m18.233867833s) [upload=57112]: Got next segment. Inline: false Jul 26 23:08:47.043614402 (59h33m18.233912982s) [upload=57112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:08:47.043620671 (59h33m18.23391925s) [upload=57112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:08:47.159920225 (59h33m18.350218792s) [seg_pos={0 0}, upload=57112]: Begin upload segment... Jul 26 23:08:47.159977395 (59h33m18.35027596s) [seg_pos={0 0}, upload=57112]: Done begin upload segment. Jul 26 23:08:47.160002448 (59h33m18.350301027s) [noise=true , address=77.90.31.12:4589 , piece_id=NCQ3USKJZFH3UCLE4YYON3RM5XPREPMKYJFYGAQA7XIUZBH52PDQ, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.160389861 (59h33m18.35068844s) [noise=true , address=45.91.116.10:4083 , piece_id=AEJGEYNRC4H65Q75GIN2JR7ML25XUBDR6DRLPCHRZ7NUNOT7Q2ZA, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.160700402 (59h33m18.350998982s) [noise=true , address=104.36.82.14:4221 , piece_id=BNW6YJBTUTQIYGBEGLOO6BRFPTFY3EMAPOZFV7BSTXJO4G62ZNIA, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.161207352 (59h33m18.351505931s) [noise=true , address=77.90.31.10:4518 , piece_id=DXF5CS37OUQIB4VJTOQBIVRMZO53CYCIZN5N6VJ66EWPIW32PR3Q, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.161329708 (59h33m18.35162829s) [noise=true , address=45.91.116.7:4009 , piece_id=EAK2T4ZGSESWZHI23E6IXV63AFPMYITEDSLGVBNPNVMCEPCOOD7A, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.161851725 (59h33m18.352150307s) [noise=true , address=104.36.82.9:4006 , piece_id=TUZIYYBPOMXPTLQ6NCP42S6KKUGIK7N7E3RZNXSFH3EDPOK5UG7A, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.162213793 (59h33m18.352512376s) [noise=true , address=77.90.31.14:4662 , piece_id=XFEHMOEXIMK7MHH2TL3HJGPFW6LIKRKO36PAW4CUN5LWWBXAHHHA, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.162643710 (59h33m18.352942292s) [noise=true , address=45.91.116.5:4298 , piece_id=G2XFSFNWRALGSE3IQGUKKLFTMFI4PZKVFMOE2ADXPRGRUVHIEANQ, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.162708525 (59h33m18.353007107s) [noise=true , address=45.91.116.14:4189 , piece_id=YRHOGSWAQ5PN2SC2DB63ECZHB22SIGCJPVS2D64C5KHI7ZPLMMNA, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.163274279 (59h33m18.353572861s) [noise=true , address=104.36.82.15:4257 , piece_id=BZRZFEQEYTW4CVJ6PAEL6YF7JJ3DIGOMGR63PCTGEUMA6RBL2ITQ, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.163335388 (59h33m18.353633966s) [noise=true , address=45.91.116.7:4010 , piece_id=7O755TAQRDFD34LOATKMMESHTQI4OEVHEJ5UW3ORALT3DS4K3UBA, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.163755464 (59h33m18.354054043s) [noise=true , address=104.36.82.11:4113 , piece_id=S2OYJQUAE2VEXMXU63ADHZNIAEZS65PMQXKLIJYEYEOQBX7ABRPQ, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.164211645 (59h33m18.354510231s) [noise=true , address=45.91.116.13:4154 , piece_id=6237K7D3SNSXKRHTFYAXNEJ6HV3X3HA62BREPVB7YNRHXLDSD36A, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.164537928 (59h33m18.354836508s) [noise=true , address=45.91.116.8:4045 , piece_id=K27P2PNSALMWXRGGTE7YGUYIWB2XY5B7IBECKWUEH4ZLQMY2JPQQ, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.164931237 (59h33m18.355229822s) [noise=true , address=104.36.82.6:4876 , piece_id=2ZVLNHUD4VC5LHZHKMUOF7PSIVGF2ARXZIE352USFAC3VAW4TB5A, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.165283431 (59h33m18.355582012s) [noise=true , address=104.36.82.2:4732 , piece_id=ZIFF5YDHNYGMMUWAF2XM3QGQNX4OHD6RQEASM2M7DX3RTHMKJU5Q, seg_pos={0 0}, upload=57112]: Uploading piece... Jul 26 23:08:47.801689766 (59h33m18.991988332s) [upload=57112]: Next returned nil segment Jul 26 23:08:47.801706468 (59h33m18.99200505s) [upload=57112]: Waiting for error group managing segments... Jul 26 23:08:47.960056267 (59h33m19.150354846s) [noise=true , address=45.91.116.10:4083 , piece_id=AEJGEYNRC4H65Q75GIN2JR7ML25XUBDR6DRLPCHRZ7NUNOT7Q2ZA, seg_pos={0 0}, upload=57112]: Done uploading piece. err: <nil> Jul 26 23:08:47.970920313 (59h33m19.161218886s) [noise=true , address=45.91.116.7:4010 , piece_id=7O755TAQRDFD34LOATKMMESHTQI4OEVHEJ5UW3ORALT3DS4K3UBA, seg_pos={0 0}, upload=57112]: Done uploading piece. err: <nil> Jul 26 23:08:47.980249615 (59h33m19.170548181s) [noise=true , address=45.91.116.8:4045 , piece_id=K27P2PNSALMWXRGGTE7YGUYIWB2XY5B7IBECKWUEH4ZLQMY2JPQQ, seg_pos={0 0}, upload=57112]: Done uploading piece. err: <nil> Jul 26 23:08:47.985171685 (59h33m19.175470251s) [noise=true , address=45.91.116.7:4009 , piece_id=EAK2T4ZGSESWZHI23E6IXV63AFPMYITEDSLGVBNPNVMCEPCOOD7A, seg_pos={0 0}, upload=57112]: Done uploading piece. err: <nil> Jul 26 23:08:47.989699500 (59h33m19.179998073s) [noise=true , address=77.90.31.10:4518 , piece_id=DXF5CS37OUQIB4VJTOQBIVRMZO53CYCIZN5N6VJ66EWPIW32PR3Q, seg_pos={0 0}, upload=57112]: Done uploading piece. err: <nil> Jul 26 23:08:47.991817801 (59h33m19.182116389s) [noise=true , address=45.91.116.5:4298 , piece_id=G2XFSFNWRALGSE3IQGUKKLFTMFI4PZKVFMOE2ADXPRGRUVHIEANQ, seg_pos={0 0}, upload=57112]: Done uploading piece. err: <nil> Jul 26 23:08:48.026591315 (59h33m19.216889884s) [noise=true , address=45.91.116.13:4154 , piece_id=6237K7D3SNSXKRHTFYAXNEJ6HV3X3HA62BREPVB7YNRHXLDSD36A, seg_pos={0 0}, upload=57112]: Done uploading piece. err: <nil> Jul 26 23:08:48.038928510 (59h33m19.229227077s) [noise=true , address=77.90.31.14:4662 , piece_id=XFEHMOEXIMK7MHH2TL3HJGPFW6LIKRKO36PAW4CUN5LWWBXAHHHA, seg_pos={0 0}, upload=57112]: Done uploading piece. err: <nil> Jul 26 23:08:48.090020446 (59h33m19.280319012s) [noise=true , address=45.91.116.14:4189 , piece_id=YRHOGSWAQ5PN2SC2DB63ECZHB22SIGCJPVS2D64C5KHI7ZPLMMNA, seg_pos={0 0}, upload=57112]: Done uploading piece. err: <nil> Jul 26 23:09:09.237364682 (59h33m40.427663267s) [noise=true , address=104.36.82.9:4006 , piece_id=TUZIYYBPOMXPTLQ6NCP42S6KKUGIK7N7E3RZNXSFH3EDPOK5UG7A, seg_pos={0 0}, upload=57112]: Done uploading piece. err: <nil> Jul 26 23:09:16.763528164 (59h33m47.953826729s) [noise=true , address=104.36.82.14:4221 , piece_id=BNW6YJBTUTQIYGBEGLOO6BRFPTFY3EMAPOZFV7BSTXJO4G62ZNIA, seg_pos={0 0}, upload=57112]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:09:16.763589895 (59h33m47.953888475s) [noise=true , address=104.36.82.6:4876 , piece_id=2ZVLNHUD4VC5LHZHKMUOF7PSIVGF2ARXZIE352USFAC3VAW4TB5A, seg_pos={0 0}, upload=57112]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:41212->104.36.82.6:4876: use of closed network connection; write tcp 139.59.144.22:41212->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41212->104.36.82.6:4876: use of closed network connection Jul 26 23:09:16.763637536 (59h33m47.953936117s) [noise=true , address=104.36.82.11:4113 , piece_id=S2OYJQUAE2VEXMXU63ADHZNIAEZS65PMQXKLIJYEYEOQBX7ABRPQ, seg_pos={0 0}, upload=57112]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:48076->104.36.82.11:4113: use of closed network connection; write tcp 139.59.144.22:48076->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48076->104.36.82.11:4113: use of closed network connection Jul 26 23:09:16.763683828 (59h33m47.953982407s) [noise=true , address=104.36.82.2:4732 , piece_id=ZIFF5YDHNYGMMUWAF2XM3QGQNX4OHD6RQEASM2M7DX3RTHMKJU5Q, seg_pos={0 0}, upload=57112]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 139.59.144.22:58026->104.36.82.2:4732: use of closed network connection; write tcp 139.59.144.22:58026->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58026->104.36.82.2:4732: use of closed network connection Jul 26 23:09:16.763724570 (59h33m47.95402315s) [noise=true , address=104.36.82.15:4257 , piece_id=BZRZFEQEYTW4CVJ6PAEL6YF7JJ3DIGOMGR63PCTGEUMA6RBL2ITQ, seg_pos={0 0}, upload=57112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:51208->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:51208->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51208->104.36.82.15:4257: use of closed network connection Jul 26 23:09:16.763889952 (59h33m47.954188527s) [noise=true , address=77.90.31.12:4589 , piece_id=NCQ3USKJZFH3UCLE4YYON3RM5XPREPMKYJFYGAQA7XIUZBH52PDQ, seg_pos={0 0}, upload=57112]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:09:16.763926218 (59h33m47.954224806s) [seg_pos={0 0}, upload=57112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:09:16.872791117 (59h33m48.063089694s) [upload=57112]: Done uploading segments... Jul 26 23:09:16.872825083 (59h33m48.063123666s) [upload=57112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.111359199s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [....XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.XX] duration: 1m0.109183341s FAIL (2025-07-26 23:08:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== GIMYXFQQUJXF6MER4IFK5MFM4FVWGQEID2IMA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.494281893 (59h32m25.684580477s) [noise=true , address=104.36.82.46:4488 , piece_id=BXZDZKONUW7KZHSDZG2W36WZZAQGRD3L7WMAHC4ATHVNYKXYMSOQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.494647291 (59h32m25.684945857s) [noise=true , address=104.36.82.55:4228 , piece_id=H5R3RO2QR2MY442KURXZJMV2VHLGUNM3VGPDBCNV2AFPC2TL7H7A, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.494680050 (59h32m25.684978632s) [noise=true , address=104.36.82.10:4092 , piece_id=BI4I6KA7FMXLZRYCOMUZKLOOCMIG5BG7P7WFOB55AF24M2HVXBJA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.494963704 (59h32m25.685262289s) [noise=true , address=104.36.82.41:4792 , piece_id=UZXN3XBIZG3RI7UML7DLGT42Q232VY35KVX5XDSG2E32PSGSP7UA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.495204799 (59h32m25.685503381s) [noise=true , address=104.36.82.52:4102 , piece_id=DJCRQB7TLRCRYGQTINCXKH4OVEDZUGL4WZQV6UHIOTWHPZT34TOQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.495450502 (59h32m25.685749081s) [noise=true , address=104.36.82.51:4082 , piece_id=JKZ66BI4FIUHOYCHE2CNB5PNB6EMM4PKM3NLVGB6MQC6CS4YPH5Q, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.495734441 (59h32m25.686033026s) [noise=true , address=104.36.82.47:4544 , piece_id=BOGWO4ZBYXNPZAWDSVSURLCGZQH6CCKWOFEHVKPICRZE6XTDQL7Q, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.496014873 (59h32m25.68631347s) [noise=true , address=104.36.82.42:4819 , piece_id=5S7YOGIGMMNUYFXDKGSHDHPQKVEUK5C4TTFB5B36ZGJ5EGJ4F4UA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.496408322 (59h32m25.686706901s) [noise=true , address=104.36.82.56:4250 , piece_id=GPOOLCCNXPBX3ANVS5TYMKIVR6FCQX2OIY2JJN4FW7VDH66ZRUIA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.496915355 (59h32m25.687213936s) [noise=true , address=104.36.82.10:4096 , piece_id=6YVLJBU4LUU7N5CW36RG2YJRSCVNWAZ36OCGDLR733PJJGIBD3YA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.497019891 (59h32m25.687318482s) [noise=true , address=79.127.219.38:4006 , piece_id=IX24VYV3A2VI4P7V7VFPSU3IB26DGK6CB6ZR5BNRIJEECA5SFLGA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.497410087 (59h32m25.687708667s) [noise=true , address=104.36.82.54:4161 , piece_id=Q4IJZW2SNBY5EM5YCNDMOROGUDT3H5IV3QY3YVYC7PD2RRYZ3OHA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.497781243 (59h32m25.688079826s) [noise=true , address=45.91.116.14:4194 , piece_id=Y3LFPXJEPRSURNBBTI2WKXXDMB5U4PFLKFT5EEMNS5V3R3CO63FQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.498058987 (59h32m25.688357565s) [noise=true , address=104.36.82.36:4624 , piece_id=IMQXVWLVQP4PLRZAZIAFJXQS5CZQOIWSV7KJYPCUGUZ2Y4M2QQHA, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.498389855 (59h32m25.688688433s) [noise=true , address=77.90.31.12:4611 , piece_id=V56NZRFEZH7SN6A2E3S5JNA3B2I5SGWF5YX4YRMSJTHXR5HEON6A, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.498668718 (59h32m25.688967296s) [noise=true , address=104.36.82.21:4482 , piece_id=UCBALDOS3OIL66RR4VKBRR6EVITG6ZF4YYO2GRJRX3L4AXUKAFXQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:54.498761422 (59h32m25.689060002s) [noise=true , address=79.127.201.215:4356 , piece_id=KAOMOYSUS2YH45BH4TVLK62A5AVFLIEZPQEPRHOSK5LVAXZB2YHQ, seg_pos={0 0}, upload=57098]: Uploading piece... Jul 26 23:07:55.074701029 (59h32m26.264999615s) [upload=57098]: Next returned nil segment Jul 26 23:07:55.074724792 (59h32m26.26502339s) [upload=57098]: Waiting for error group managing segments... Jul 26 23:07:55.216394130 (59h32m26.4066927s) [noise=true , address=45.91.116.14:4194 , piece_id=Y3LFPXJEPRSURNBBTI2WKXXDMB5U4PFLKFT5EEMNS5V3R3CO63FQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.231153209 (59h32m26.421451775s) [noise=true , address=104.36.82.52:4102 , piece_id=DJCRQB7TLRCRYGQTINCXKH4OVEDZUGL4WZQV6UHIOTWHPZT34TOQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.235588310 (59h32m26.425886883s) [noise=true , address=104.36.82.54:4161 , piece_id=Q4IJZW2SNBY5EM5YCNDMOROGUDT3H5IV3QY3YVYC7PD2RRYZ3OHA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.236945932 (59h32m26.427244504s) [noise=true , address=104.36.82.55:4228 , piece_id=H5R3RO2QR2MY442KURXZJMV2VHLGUNM3VGPDBCNV2AFPC2TL7H7A, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.252295061 (59h32m26.44259363s) [noise=true , address=104.36.82.56:4250 , piece_id=GPOOLCCNXPBX3ANVS5TYMKIVR6FCQX2OIY2JJN4FW7VDH66ZRUIA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:55.314244807 (59h32m26.504543373s) [noise=true , address=79.127.219.38:4006 , piece_id=IX24VYV3A2VI4P7V7VFPSU3IB26DGK6CB6ZR5BNRIJEECA5SFLGA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:07:58.098342640 (59h32m29.288641211s) [noise=true , address=104.36.82.47:4544 , piece_id=BOGWO4ZBYXNPZAWDSVSURLCGZQH6CCKWOFEHVKPICRZE6XTDQL7Q, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:01.042437350 (59h32m32.232735935s) [noise=true , address=104.36.82.46:4488 , piece_id=BXZDZKONUW7KZHSDZG2W36WZZAQGRD3L7WMAHC4ATHVNYKXYMSOQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:04.364883316 (59h32m35.555181895s) [noise=true , address=79.127.201.215:4356 , piece_id=KAOMOYSUS2YH45BH4TVLK62A5AVFLIEZPQEPRHOSK5LVAXZB2YHQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:08.994662985 (59h32m40.184961558s) [noise=true , address=79.127.201.215:4355 , piece_id=725MUVVECDVEHOHGIMYXFQQUJXF6MER4IFK5MFM4FVWGQEID2IMA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:44.699576901 (59h33m15.889875474s) [noise=true , address=104.36.82.10:4092 , piece_id=BI4I6KA7FMXLZRYCOMUZKLOOCMIG5BG7P7WFOB55AF24M2HVXBJA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: <nil> Jul 26 23:08:53.938251203 (59h33m25.128549772s) [noise=true , address=104.36.82.51:4082 , piece_id=JKZ66BI4FIUHOYCHE2CNB5PNB6EMM4PKM3NLVGB6MQC6CS4YPH5Q, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:1m5NjTfYFQoyaftkARd75HMFZDW1jpnVj7H8LaUZWZcZYozHru, address:104.36.82.51:4082): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:08:53.938467746 (59h33m25.128766316s) [noise=true , address=104.36.82.10:4096 , piece_id=6YVLJBU4LUU7N5CW36RG2YJRSCVNWAZ36OCGDLR733PJJGIBD3YA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA, address:104.36.82.10:4096): protocol: write tcp 139.59.144.22:37300->104.36.82.10:4096: use of closed network connection; write tcp 139.59.144.22:37300->104.36.82.10:4096: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37300->104.36.82.10:4096: use of closed network connection Jul 26 23:08:53.938546369 (59h33m25.128844952s) [noise=true , address=104.36.82.21:4482 , piece_id=UCBALDOS3OIL66RR4VKBRR6EVITG6ZF4YYO2GRJRX3L4AXUKAFXQ, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:12B2s428CMVK7mdLV84VgWFcdRQaVQxcA7YuqqeqEM8FAefEGt7, address:104.36.82.21:4482): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:08:53.938573528 (59h33m25.128872098s) [noise=true , address=104.36.82.36:4624 , piece_id=IMQXVWLVQP4PLRZAZIAFJXQS5CZQOIWSV7KJYPCUGUZ2Y4M2QQHA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1, address:104.36.82.36:4624): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:08:53.938624398 (59h33m25.128922967s) [noise=true , address=77.90.31.12:4611 , piece_id=V56NZRFEZH7SN6A2E3S5JNA3B2I5SGWF5YX4YRMSJTHXR5HEON6A, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:1r7qdqiUuUqsKnTXdC3S1Ek84rsCPtjPoXbPugVHWBzJezv4W2, address:77.90.31.12:4611): protocol: rpc: dial tcp 77.90.31.12:4611: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:53.938673213 (59h33m25.128971786s) [noise=true , address=104.36.82.41:4792 , piece_id=UZXN3XBIZG3RI7UML7DLGT42Q232VY35KVX5XDSG2E32PSGSP7UA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k, address:104.36.82.41:4792): protocol: write tcp 139.59.144.22:45786->104.36.82.41:4792: use of closed network connection; write tcp 139.59.144.22:45786->104.36.82.41:4792: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45786->104.36.82.41:4792: use of closed network connection Jul 26 23:08:53.938718991 (59h33m25.129017561s) [noise=true , address=104.36.82.42:4819 , piece_id=5S7YOGIGMMNUYFXDKGSHDHPQKVEUK5C4TTFB5B36ZGJ5EGJ4F4UA, seg_pos={0 0}, upload=57098]: Done uploading piece. err: ecclient: upload failed (node:12MZhrDu1fjMN7XFoR2h1q3JPLHJYYURGQwQgcgLuucSrGShjoS, address:104.36.82.42:4819): protocol: write tcp 139.59.144.22:36980->104.36.82.42:4819: use of closed network connection; write tcp 139.59.144.22:36980->104.36.82.42:4819: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36980->104.36.82.42:4819: use of closed network connection Jul 26 23:08:53.938738074 (59h33m25.129036655s) [seg_pos={0 0}, upload=57098]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:08:54.045356315 (59h33m25.235654887s) [upload=57098]: Done uploading segments... Jul 26 23:08:54.045383269 (59h33m25.235681848s) [upload=57098]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.109199542s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...X..X...X.XX...X..X.XX......X.X.......X.......X........XX.] duration: 13.218357711s FAIL (2025-07-26 23:08:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.219156645s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 30.001161925s FAIL (2025-07-26 23:08:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001197737s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [....X..X...X.XX...X..X.XX......X.X.......X.......X........XX] duration: 1m0.000726352s FAIL (2025-07-26 23:08:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00074856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................X......XXX] duration: 30.107871302s FAIL (2025-07-26 23:08:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== id=7QEMGN6YCQB7S4MOYWTCRTUWTGCBJXKXD7CGU3IOFHVMPYLPWCZA, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.731047006 (59h32m26.921345592s) [noise=true , address=104.36.82.9:4007 , piece_id=DSH4UOQPH7FEX4I2DYRFGX7IGLRMXZS6EL3BPRSDLHAOOAJ5RJGA, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.731289083 (59h32m26.921587662s) [noise=true , address=104.36.82.2:4730 , piece_id=ASN2VU3EOYXYR6PW6M4HTJ7II77UJMQTXVZKXRUJIRIJ7WAJY4YQ, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.731525647 (59h32m26.921824225s) [noise=true , address=104.36.82.4:4801 , piece_id=2363XQ7FNF7FJPD76BAFOIEXZWTETDQJY4WL6CROBACP2LZUJBSQ, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.731758728 (59h32m26.922057319s) [noise=true , address=45.91.116.13:4155 , piece_id=5QCRJ5EOYJYDQNNAC4L6UFRXOD3WSIWFJFUICLKA55TF5POFJNUA, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.732081116 (59h32m26.922379696s) [noise=true , address=45.91.116.4:4940 , piece_id=USZ4TEE4NGHE74DXUB3VSZT67Z55GUYHWZAFFZYBHKRTLMEKTZAQ, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.732380977 (59h32m26.922679564s) [noise=true , address=45.91.116.5:4297 , piece_id=W3GG6WTGBGAC53FTVKDEJ4EEMYNQII2TS6SCLU2N525RHRIDEPOQ, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.732747138 (59h32m26.923045719s) [noise=true , address=45.91.116.10:4083 , piece_id=JGNHB2ER6KWRBBAIQOKG6IN5RJ6LQJAROZYVK4M6OQVM6BL3K3LA, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.733099706 (59h32m26.923398295s) [noise=true , address=104.36.82.14:4222 , piece_id=S3DU4I5TD43JYNZBJMMZY6CEIPTHLME4OSPE4KGPBUAAEO6DCW4A, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.733437128 (59h32m26.923735707s) [noise=true , address=45.91.116.13:4154 , piece_id=5UPM5KSDSFKQNAODNHQIBZDTG4KPDLIABAO55QXLDT5HG73JATYA, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.733729599 (59h32m26.924028178s) [noise=true , address=45.91.116.12:4227 , piece_id=4LRT3HWZFFQEV7DWPBULNEBTDVXCHA54EPJXKJHCZZBOLAVXRSXQ, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.734142201 (59h32m26.92444078s) [noise=true , address=104.36.82.6:4873 , piece_id=I6PZ7K7RRDXAS4ZROYDGHMKT4NME4J5KJGMEFFPAZVJZHLLGRGMA, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.734640955 (59h32m26.92493953s) [noise=true , address=104.36.82.12:4151 , piece_id=CT73EOBZYIMVQ5YSQHES7B4QU54ZQTDN75DTHAA2LSLZCUOF3G2Q, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.735160906 (59h32m26.925459489s) [noise=true , address=104.36.82.2:4731 , piece_id=73EDOPO5M46CDTQG53OENCO7OXC5GJ4MUES6WPX2BRIMIXFDDSNQ, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.735904509 (59h32m26.926203091s) [noise=true , address=104.36.82.8:4969 , piece_id=LHCXDU3PHOYELBDMZMFJWWKFG6EWWQETZT7B5DJRNSLKZL2ICVOQ, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:55.736256704 (59h32m26.926555286s) [noise=true , address=77.90.31.19:4842 , piece_id=QLKLUFKXCTHB7TT6A6NHH43QL3Q4HVR4LHFMMTZBMXOV4B66VQ5A, seg_pos={0 0}, upload=57100]: Uploading piece... Jul 26 23:07:56.351360382 (59h32m27.541658951s) [upload=57100]: Next returned nil segment Jul 26 23:07:56.351379113 (59h32m27.541677692s) [upload=57100]: Waiting for error group managing segments... Jul 26 23:07:56.489828552 (59h32m27.680127121s) [noise=true , address=45.91.116.12:4227 , piece_id=4LRT3HWZFFQEV7DWPBULNEBTDVXCHA54EPJXKJHCZZBOLAVXRSXQ, seg_pos={0 0}, upload=57100]: Done uploading piece. err: <nil> Jul 26 23:07:56.490763393 (59h32m27.681061974s) [noise=true , address=77.90.31.19:4842 , piece_id=QLKLUFKXCTHB7TT6A6NHH43QL3Q4HVR4LHFMMTZBMXOV4B66VQ5A, seg_pos={0 0}, upload=57100]: Done uploading piece. err: <nil> Jul 26 23:07:56.496199680 (59h32m27.686498254s) [noise=true , address=45.91.116.10:4083 , piece_id=JGNHB2ER6KWRBBAIQOKG6IN5RJ6LQJAROZYVK4M6OQVM6BL3K3LA, seg_pos={0 0}, upload=57100]: Done uploading piece. err: <nil> Jul 26 23:07:56.502392483 (59h32m27.692691053s) [noise=true , address=45.91.116.4:4940 , piece_id=USZ4TEE4NGHE74DXUB3VSZT67Z55GUYHWZAFFZYBHKRTLMEKTZAQ, seg_pos={0 0}, upload=57100]: Done uploading piece. err: <nil> Jul 26 23:07:56.502563134 (59h32m27.692861703s) [noise=true , address=45.91.116.13:4154 , piece_id=5UPM5KSDSFKQNAODNHQIBZDTG4KPDLIABAO55QXLDT5HG73JATYA, seg_pos={0 0}, upload=57100]: Done uploading piece. err: <nil> Jul 26 23:07:56.502848673 (59h32m27.693147246s) [noise=true , address=45.91.116.13:4155 , piece_id=5QCRJ5EOYJYDQNNAC4L6UFRXOD3WSIWFJFUICLKA55TF5POFJNUA, seg_pos={0 0}, upload=57100]: Done uploading piece. err: <nil> Jul 26 23:07:56.521833455 (59h32m27.712132038s) [noise=true , address=45.91.116.5:4297 , piece_id=W3GG6WTGBGAC53FTVKDEJ4EEMYNQII2TS6SCLU2N525RHRIDEPOQ, seg_pos={0 0}, upload=57100]: Done uploading piece. err: <nil> Jul 26 23:07:56.537673253 (59h32m27.727971823s) [noise=true , address=77.90.31.18:4807 , piece_id=7QEMGN6YCQB7S4MOYWTCRTUWTGCBJXKXD7CGU3IOFHVMPYLPWCZA, seg_pos={0 0}, upload=57100]: Done uploading piece. err: <nil> Jul 26 23:08:25.348736881 (59h32m56.539035482s) [noise=true , address=104.36.82.14:4222 , piece_id=S3DU4I5TD43JYNZBJMMZY6CEIPTHLME4OSPE4KGPBUAAEO6DCW4A, seg_pos={0 0}, upload=57100]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:33644->104.36.82.14:4222: use of closed network connection; write tcp 139.59.144.22:33644->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33644->104.36.82.14:4222: use of closed network connection Jul 26 23:08:25.348754801 (59h32m56.539053374s) [noise=true , address=104.36.82.9:4007 , piece_id=DSH4UOQPH7FEX4I2DYRFGX7IGLRMXZS6EL3BPRSDLHAOOAJ5RJGA, seg_pos={0 0}, upload=57100]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:40250->104.36.82.9:4007: use of closed network connection; write tcp 139.59.144.22:40250->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40250->104.36.82.9:4007: use of closed network connection Jul 26 23:08:25.348850719 (59h32m56.539149304s) [noise=true , address=104.36.82.6:4873 , piece_id=I6PZ7K7RRDXAS4ZROYDGHMKT4NME4J5KJGMEFFPAZVJZHLLGRGMA, seg_pos={0 0}, upload=57100]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 139.59.144.22:43430->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:43430->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43430->104.36.82.6:4873: use of closed network connection Jul 26 23:08:25.348926107 (59h32m56.539224672s) [noise=true , address=104.36.82.2:4731 , piece_id=73EDOPO5M46CDTQG53OENCO7OXC5GJ4MUES6WPX2BRIMIXFDDSNQ, seg_pos={0 0}, upload=57100]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:47506->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:47506->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47506->104.36.82.2:4731: use of closed network connection Jul 26 23:08:25.349051602 (59h32m56.53935018s) [noise=true , address=104.36.82.4:4801 , piece_id=2363XQ7FNF7FJPD76BAFOIEXZWTETDQJY4WL6CROBACP2LZUJBSQ, seg_pos={0 0}, upload=57100]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:38590->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:38590->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38590->104.36.82.4:4801: use of closed network connection Jul 26 23:08:25.349109034 (59h32m56.539407614s) [noise=true , address=104.36.82.12:4151 , piece_id=CT73EOBZYIMVQ5YSQHES7B4QU54ZQTDN75DTHAA2LSLZCUOF3G2Q, seg_pos={0 0}, upload=57100]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: read tcp 139.59.144.22:45144->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:25.349118586 (59h32m56.539417159s) [noise=true , address=104.36.82.8:4969 , piece_id=LHCXDU3PHOYELBDMZMFJWWKFG6EWWQETZT7B5DJRNSLKZL2ICVOQ, seg_pos={0 0}, upload=57100]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: read tcp 139.59.144.22:59024->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:25.349188389 (59h32m56.539486957s) [noise=true , address=104.36.82.2:4730 , piece_id=ASN2VU3EOYXYR6PW6M4HTJ7II77UJMQTXVZKXRUJIRIJ7WAJY4YQ, seg_pos={0 0}, upload=57100]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: read tcp 139.59.144.22:36730->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:25.349223626 (59h32m56.539522205s) [seg_pos={0 0}, upload=57100]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:08:25.454120280 (59h32m56.64441886s) [upload=57100]: Done uploading segments... Jul 26 23:08:25.454147285 (59h32m56.644445865s) [upload=57100]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.107896653s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.............................X..................X.....XXXX] duration: 30.119592302s FAIL (2025-07-26 23:08:25 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.119616078s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [.....XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.X] duration: 1m0.103096076s FAIL (2025-07-26 23:07:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 31m25.354539368s) [noise=true , address=104.36.82.44:4874 , piece_id=DPGBYMMODN6CAADRFCGVQAD3UQX36BFTY6LJAFVCDKYS64CSI64A, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.165369151 (59h31m25.35566774s) [noise=true , address=104.36.82.22:4532 , piece_id=S5RGNPR6672ZLJ3NKW57FSIGGHNBAYSPDOSUNZFRUF62XLXXERDQ, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.166524776 (59h31m25.356823365s) [noise=true , address=79.127.205.233:4624 , piece_id=V7AWTEO6RQAQSNV36FGQALGMDXDWXP4EPBPGDXEDI7HVCUHO6DZA, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.170154804 (59h31m25.360453404s) [noise=true , address=79.127.205.235:4679 , piece_id=ABXWALUL7DNSFFUQVJGUQEZ54WH5GENOMW6PLEFXHY7NRI7FFPBA, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.170669557 (59h31m25.360968142s) [noise=true , address=104.36.82.7:4943 , piece_id=2OU32EMB6CVOKR5JOL7OWGFBBW6KL4PJL6J22TXCCXJQZSV6JO3A, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.171042610 (59h31m25.361341198s) [noise=true , address=104.36.82.54:4187 , piece_id=NIDEIWEIAMIYNAKNBWINR5V7BLC6IVMTHP4WCZCXNIMMY3GO46RQ, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.171463890 (59h31m25.361762472s) [noise=true , address=104.36.82.36:4650 , piece_id=RSRXQSKBL42GA3AAU7TLZSVBKXK7LAG25RUM572X5XKG4GPDAFVA, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.171836907 (59h31m25.362135492s) [noise=true , address=104.36.82.53:4135 , piece_id=2E7RDVEZAQUKHRWENFHKOG7WKK7M7PCCPOG7RTLTD7ARJPBEC7DA, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.172202031 (59h31m25.362500612s) [noise=true , address=104.36.82.40:4954 , piece_id=KETR6JHKUNRMSLJEDPBARBEUAUMGJCU2GDMO7OQ57XD5HAMQYCMA, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.172670470 (59h31m25.36296904s) [noise=true , address=104.36.82.47:4550 , piece_id=234ZGDNA6QED6KLWMZLVRW7P4NDAPKH66KLHWPL4AULBWNUIA77A, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.174151345 (59h31m25.364449933s) [noise=true , address=104.36.82.49:4997 , piece_id=KCZ7RANFLXW6A7C54CMK64FKSAAP3BFTQL5Q25RXERJT25Y2RY5A, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.174540299 (59h31m25.364838878s) [noise=true , address=104.36.82.46:4494 , piece_id=66KQ75E7AYD43OXY3ECQ2G6U4KIEM5W5Q67LYIUPIVHLQNSKWHRQ, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.174888037 (59h31m25.365186617s) [noise=true , address=104.36.82.56:4258 , piece_id=3UNGYMHHWZSQVI4FNJA2QZOJH5ME6DYNYAT5ULO6QSW5DZQB3UHA, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.175226468 (59h31m25.365525048s) [noise=true , address=104.36.82.41:4773 , piece_id=PDHXGOXH6TDWVEXZF7HAUWNZ6RWYFFRLMGNIYMJE2W2OMA6VXGNA, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.175603937 (59h31m25.365902519s) [noise=true , address=79.127.201.215:4355 , piece_id=VLHDVC6LTSBKLINPRT45STKM33VD6P575RRWSM6HLKKPUMNLCYYA, seg_pos={0 0}, upload=57080]: Uploading piece... Jul 26 23:06:54.790738405 (59h31m25.981036975s) [upload=57080]: Next returned nil segment Jul 26 23:06:54.790756989 (59h31m25.981055558s) [upload=57080]: Waiting for error group managing segments... Jul 26 23:06:54.934681832 (59h31m26.124980412s) [noise=true , address=79.127.219.38:4017 , piece_id=SCJUSDXWK4MFXD33QGFDIBLJUHOYRZ4QK5ZDOEVLUYBOH7WYYGVQ, seg_pos={0 0}, upload=57080]: Done uploading piece. err: <nil> Jul 26 23:06:55.245045691 (59h31m26.43534426s) [noise=true , address=79.127.201.215:4355 , piece_id=VLHDVC6LTSBKLINPRT45STKM33VD6P575RRWSM6HLKKPUMNLCYYA, seg_pos={0 0}, upload=57080]: Done uploading piece. err: <nil> Jul 26 23:06:55.457863920 (59h31m26.648162491s) [noise=true , address=79.127.205.235:4679 , piece_id=ABXWALUL7DNSFFUQVJGUQEZ54WH5GENOMW6PLEFXHY7NRI7FFPBA, seg_pos={0 0}, upload=57080]: Done uploading piece. err: <nil> Jul 26 23:06:55.517902344 (59h31m26.708200931s) [noise=true , address=79.127.205.233:4624 , piece_id=V7AWTEO6RQAQSNV36FGQALGMDXDWXP4EPBPGDXEDI7HVCUHO6DZA, seg_pos={0 0}, upload=57080]: Done uploading piece. err: <nil> Jul 26 23:06:58.226108837 (59h31m29.416407406s) [noise=true , address=104.36.82.53:4135 , piece_id=2E7RDVEZAQUKHRWENFHKOG7WKK7M7PCCPOG7RTLTD7ARJPBEC7DA, seg_pos={0 0}, upload=57080]: Done uploading piece. err: <nil> Jul 26 23:06:58.289185799 (59h31m29.47948437s) [noise=true , address=104.36.82.56:4258 , piece_id=3UNGYMHHWZSQVI4FNJA2QZOJH5ME6DYNYAT5ULO6QSW5DZQB3UHA, seg_pos={0 0}, upload=57080]: Done uploading piece. err: <nil> Jul 26 23:07:02.185480318 (59h31m33.37577889s) [noise=true , address=104.36.82.54:4187 , piece_id=NIDEIWEIAMIYNAKNBWINR5V7BLC6IVMTHP4WCZCXNIMMY3GO46RQ, seg_pos={0 0}, upload=57080]: Done uploading piece. err: <nil> Jul 26 23:07:11.714066762 (59h31m42.904365336s) [noise=true , address=104.36.82.47:4550 , piece_id=234ZGDNA6QED6KLWMZLVRW7P4NDAPKH66KLHWPL4AULBWNUIA77A, seg_pos={0 0}, upload=57080]: Done uploading piece. err: <nil> Jul 26 23:07:21.256065895 (59h31m52.446364467s) [noise=true , address=104.36.82.46:4494 , piece_id=66KQ75E7AYD43OXY3ECQ2G6U4KIEM5W5Q67LYIUPIVHLQNSKWHRQ, seg_pos={0 0}, upload=57080]: Done uploading piece. err: <nil> Jul 26 23:07:25.693781498 (59h31m56.88408008s) [noise=true , address=104.36.82.36:4650 , piece_id=RSRXQSKBL42GA3AAU7TLZSVBKXK7LAG25RUM572X5XKG4GPDAFVA, seg_pos={0 0}, upload=57080]: Done uploading piece. err: <nil> Jul 26 23:07:53.677026601 (59h32m24.867325189s) [noise=true , address=104.36.82.7:4943 , piece_id=2OU32EMB6CVOKR5JOL7OWGFBBW6KL4PJL6J22TXCCXJQZSV6JO3A, seg_pos={0 0}, upload=57080]: Done uploading piece. err: ecclient: upload failed (node:12BdCdPHn6ko7nfY4sPuc4NutXABzJDdHHhtSnyXri5gofhZEZJ, address:104.36.82.7:4943): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:07:53.677219051 (59h32m24.867517621s) [noise=true , address=104.36.82.5:4856 , piece_id=WJYHFW7W7BSJ2D4NRDTB6FZQ7U6KNS6BA7BXLM5JFLAD76YIPJBA, seg_pos={0 0}, upload=57080]: Done uploading piece. err: ecclient: upload failed (node:12mXTGa9rxx8jU8oKLmR9obX3Utn7p5rQH2GeK6LeJ6jiNvfYZ6, address:104.36.82.5:4856): protocol: expected piece hash; write tcp 139.59.144.22:54898->104.36.82.5:4856: use of closed network connection; write tcp 139.59.144.22:54898->104.36.82.5:4856: use of closed network connection; EOF Jul 26 23:07:53.677308411 (59h32m24.86760698s) [noise=true , address=104.36.82.44:4874 , piece_id=DPGBYMMODN6CAADRFCGVQAD3UQX36BFTY6LJAFVCDKYS64CSI64A, seg_pos={0 0}, upload=57080]: Done uploading piece. err: ecclient: upload failed (node:12YApHMARaC7Wxeqopt72gpnyJS1xYBwhV7wbU9NJdR6W7hukyq, address:104.36.82.44:4874): protocol: write tcp 139.59.144.22:38756->104.36.82.44:4874: use of closed network connection; write tcp 139.59.144.22:38756->104.36.82.44:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38756->104.36.82.44:4874: use of closed network connection Jul 26 23:07:53.677379063 (59h32m24.867677632s) [noise=true , address=104.36.82.49:4997 , piece_id=KCZ7RANFLXW6A7C54CMK64FKSAAP3BFTQL5Q25RXERJT25Y2RY5A, seg_pos={0 0}, upload=57080]: Done uploading piece. err: ecclient: upload failed (node:1qWXaiqvSg5sDEQQXgQvHFTu69JFY6QbUyKaepBoyZMzkPfS49, address:104.36.82.49:4997): protocol: write tcp 139.59.144.22:33442->104.36.82.49:4997: use of closed network connection; write tcp 139.59.144.22:33442->104.36.82.49:4997: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33442->104.36.82.49:4997: use of closed network connection Jul 26 23:07:53.677593895 (59h32m24.867892473s) [noise=true , address=104.36.82.40:4954 , piece_id=KETR6JHKUNRMSLJEDPBARBEUAUMGJCU2GDMO7OQ57XD5HAMQYCMA, seg_pos={0 0}, upload=57080]: Done uploading piece. err: ecclient: upload failed (node:14F5pJPTejP7BMpo1tcZo6uG9XdxvT7gLAUb1cgeBzKChafSXz, address:104.36.82.40:4954): protocol: read tcp 139.59.144.22:55796->104.36.82.40:4954: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:53.677637125 (59h32m24.867935705s) [noise=true , address=104.36.82.21:4487 , piece_id=S4T3SKC3TONJL4ALQPYPDOZHA4QAJGPENIUCTIXXMISSSZNXFOOQ, seg_pos={0 0}, upload=57080]: Done uploading piece. err: ecclient: upload failed (node:19xsJiapoS3bYRaeQoZ3w6U9eKWTkGpncNPifT9MAqnxCeL7yK, address:104.36.82.21:4487): protocol: read tcp 139.59.144.22:60014->104.36.82.21:4487: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:53.677662332 (59h32m24.867960915s) [noise=true , address=104.36.82.41:4773 , piece_id=PDHXGOXH6TDWVEXZF7HAUWNZ6RWYFFRLMGNIYMJE2W2OMA6VXGNA, seg_pos={0 0}, upload=57080]: Done uploading piece. err: ecclient: upload failed (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV, address:104.36.82.41:4773): protocol: write tcp 139.59.144.22:32886->104.36.82.41:4773: use of closed network connection; write tcp 139.59.144.22:32886->104.36.82.41:4773: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32886->104.36.82.41:4773: use of closed network connection Jul 26 23:07:53.677724989 (59h32m24.868023569s) [noise=true , address=104.36.82.22:4532 , piece_id=S5RGNPR6672ZLJ3NKW57FSIGGHNBAYSPDOSUNZFRUF62XLXXERDQ, seg_pos={0 0}, upload=57080]: Done uploading piece. err: ecclient: upload failed (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp, address:104.36.82.22:4532): protocol: read tcp 139.59.144.22:45448->104.36.82.22:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:53.677755743 (59h32m24.868054322s) [seg_pos={0 0}, upload=57080]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:07:53.778819526 (59h32m24.969118097s) [upload=57080]: Done uploading segments... Jul 26 23:07:53.778847257 (59h32m24.969145837s) [upload=57080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.103124911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.............................X..................X.....XXX] duration: 30.106638934s FAIL (2025-07-26 23:07:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.106657733s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.686388386s FAIL (2025-07-26 23:07:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.686407986s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001490405s FAIL (2025-07-26 23:07:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001522627s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.....X..X...X.XX...X..X.XX......X.X.......X.......X........X] duration: 1m0.001253315s FAIL (2025-07-26 23:07:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001275649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.............................X..................X.....XX] duration: 30.12972242s FAIL (2025-07-26 23:07:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.129761363s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X......XX] duration: 30.112741413s FAIL (2025-07-26 23:07:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 0896s) [upload=57077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:06:44.659422100 (59h31m15.849720678s) [seg_pos={0 0}, upload=57077]: Begin upload segment... Jul 26 23:06:44.659484992 (59h31m15.849783641s) [seg_pos={0 0}, upload=57077]: Done begin upload segment. Jul 26 23:06:44.659511188 (59h31m15.849809776s) [noise=true , address=45.91.116.13:4155 , piece_id=M3MEY275D24BLIKYIOWFD4RETO2OBKIXZ5V4H6PIUB5Q3PX64EJQ, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.659538756 (59h31m15.849837346s) [noise=true , address=45.91.116.4:4937 , piece_id=AZ6P5MZS54VIA3IND4ULPZI2ECH5ZAGM4R6VYOLRSQIFMI3NWUEQ, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.659959170 (59h31m15.850257757s) [noise=true , address=77.90.31.14:4663 , piece_id=76BE4Q3MXGESQHUDQPTXZMB7BVDD2BBNCYCUWDF6FQX6N4NI5SJQ, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.660020775 (59h31m15.850319363s) [noise=true , address=104.36.82.12:4150 , piece_id=LEGJEJ5KJFOI7LZLJAPDUS6NOZZ4G37VWKHHUPJ7XEH5YLFDW2IQ, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.660297119 (59h31m15.850595698s) [noise=true , address=45.91.116.8:4047 , piece_id=BUFT5FQGOXF3KANA6VDZHPIAQG4JUEPIFIRPZQ2RUM2LZQZNXNRQ, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.660375213 (59h31m15.850673796s) [noise=true , address=45.91.116.10:4081 , piece_id=GTKL4CEHELC7VA2OSHPWABLH56IM2M5JS635SONA4TEJHH63ZR3Q, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.660724026 (59h31m15.851022614s) [noise=true , address=104.36.82.4:4804 , piece_id=3OOX6YLC3AW5EGIH3YFQNAQWIRBFLBVBTZBRXT67YDT2SAC5T5XA, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.660912896 (59h31m15.851211475s) [noise=true , address=104.36.82.6:4873 , piece_id=MR3FXTTZF33BO7AAFM4NB5AHWVTI6DZP2VAG2JQKXY4XUZLBGLFQ, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.661028565 (59h31m15.851327147s) [noise=true , address=45.91.116.11:4119 , piece_id=YHHJDZHZYXCOIY6DCAVGDQO7FR3QU34XIW43KS55TNOU7P73V2AQ, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.661323000 (59h31m15.851621583s) [noise=true , address=77.90.31.14:4661 , piece_id=37OBTJPMXRUHIQD7YK3UBHRUEKBWPFMHNHJ4ARZI3S2CBKLVAYRA, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.661563189 (59h31m15.85186177s) [noise=true , address=104.36.82.11:4115 , piece_id=R42OSD7JPJURDJTSEUAFSPBGEOVN7G7ABYQRUBF4DLINTCM33QIA, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.661717271 (59h31m15.852015855s) [noise=true , address=104.36.82.15:4259 , piece_id=DOXQ4UVYFBSRTHQ2YR2WB6O4AYXNVZGFCCP6B6SBBUR3QGDAVLRQ, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.662107783 (59h31m15.852406374s) [noise=true , address=104.36.82.9:4005 , piece_id=ZPDCSXT2O535DHZSHDLAFGIEBH5IESNRHOEU6RKQ5RRXGZHRQPNA, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.662514717 (59h31m15.852813301s) [noise=true , address=45.91.116.7:4011 , piece_id=7VAKROKUKLA5CLQX274CN6L5UW66PF3WOIVOFTWEJC6SPCTESOKQ, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.663264786 (59h31m15.853563375s) [noise=true , address=45.91.116.5:4298 , piece_id=FSCDCHCAC7VMVQOZ7VPVIO5BSYCT2OMTOMZN4N6IB56PSXJYXSHQ, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:44.663618538 (59h31m15.85391712s) [noise=true , address=104.36.82.15:4257 , piece_id=7NFK2UPBCXRQMXNL45B3FUYULBDTVF3VAEK3N3BASZFIHSJJBWIQ, seg_pos={0 0}, upload=57077]: Uploading piece... Jul 26 23:06:45.274775931 (59h31m16.465074505s) [upload=57077]: Next returned nil segment Jul 26 23:06:45.274796950 (59h31m16.465095533s) [upload=57077]: Waiting for error group managing segments... Jul 26 23:06:45.411135286 (59h31m16.601433857s) [noise=true , address=45.91.116.10:4081 , piece_id=GTKL4CEHELC7VA2OSHPWABLH56IM2M5JS635SONA4TEJHH63ZR3Q, seg_pos={0 0}, upload=57077]: Done uploading piece. err: <nil> Jul 26 23:06:45.413539320 (59h31m16.603837887s) [noise=true , address=45.91.116.13:4155 , piece_id=M3MEY275D24BLIKYIOWFD4RETO2OBKIXZ5V4H6PIUB5Q3PX64EJQ, seg_pos={0 0}, upload=57077]: Done uploading piece. err: <nil> Jul 26 23:06:45.414371824 (59h31m16.604670391s) [noise=true , address=45.91.116.7:4011 , piece_id=7VAKROKUKLA5CLQX274CN6L5UW66PF3WOIVOFTWEJC6SPCTESOKQ, seg_pos={0 0}, upload=57077]: Done uploading piece. err: <nil> Jul 26 23:06:45.414603720 (59h31m16.604902289s) [noise=true , address=77.90.31.14:4661 , piece_id=37OBTJPMXRUHIQD7YK3UBHRUEKBWPFMHNHJ4ARZI3S2CBKLVAYRA, seg_pos={0 0}, upload=57077]: Done uploading piece. err: <nil> Jul 26 23:06:45.415780920 (59h31m16.60607949s) [noise=true , address=45.91.116.4:4937 , piece_id=AZ6P5MZS54VIA3IND4ULPZI2ECH5ZAGM4R6VYOLRSQIFMI3NWUEQ, seg_pos={0 0}, upload=57077]: Done uploading piece. err: <nil> Jul 26 23:06:45.419010716 (59h31m16.609309285s) [noise=true , address=45.91.116.8:4047 , piece_id=BUFT5FQGOXF3KANA6VDZHPIAQG4JUEPIFIRPZQ2RUM2LZQZNXNRQ, seg_pos={0 0}, upload=57077]: Done uploading piece. err: <nil> Jul 26 23:06:45.426575808 (59h31m16.616874388s) [noise=true , address=45.91.116.5:4298 , piece_id=FSCDCHCAC7VMVQOZ7VPVIO5BSYCT2OMTOMZN4N6IB56PSXJYXSHQ, seg_pos={0 0}, upload=57077]: Done uploading piece. err: <nil> Jul 26 23:06:45.431008346 (59h31m16.621306916s) [noise=true , address=77.90.31.14:4663 , piece_id=76BE4Q3MXGESQHUDQPTXZMB7BVDD2BBNCYCUWDF6FQX6N4NI5SJQ, seg_pos={0 0}, upload=57077]: Done uploading piece. err: <nil> Jul 26 23:06:45.439824477 (59h31m16.630123043s) [noise=true , address=45.91.116.11:4119 , piece_id=YHHJDZHZYXCOIY6DCAVGDQO7FR3QU34XIW43KS55TNOU7P73V2AQ, seg_pos={0 0}, upload=57077]: Done uploading piece. err: <nil> Jul 26 23:07:14.243932701 (59h31m45.434231288s) [noise=true , address=104.36.82.9:4005 , piece_id=ZPDCSXT2O535DHZSHDLAFGIEBH5IESNRHOEU6RKQ5RRXGZHRQPNA, seg_pos={0 0}, upload=57077]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 139.59.144.22:58056->104.36.82.9:4005: use of closed network connection; write tcp 139.59.144.22:58056->104.36.82.9:4005: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58056->104.36.82.9:4005: use of closed network connection Jul 26 23:07:14.244152051 (59h31m45.434450634s) [noise=true , address=104.36.82.6:4873 , piece_id=MR3FXTTZF33BO7AAFM4NB5AHWVTI6DZP2VAG2JQKXY4XUZLBGLFQ, seg_pos={0 0}, upload=57077]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 139.59.144.22:60844->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:60844->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60844->104.36.82.6:4873: use of closed network connection Jul 26 23:07:14.244231262 (59h31m45.434529848s) [noise=true , address=104.36.82.15:4257 , piece_id=7NFK2UPBCXRQMXNL45B3FUYULBDTVF3VAEK3N3BASZFIHSJJBWIQ, seg_pos={0 0}, upload=57077]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:58196->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:58196->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58196->104.36.82.15:4257: use of closed network connection Jul 26 23:07:14.244463790 (59h31m45.43476238s) [noise=true , address=104.36.82.11:4115 , piece_id=R42OSD7JPJURDJTSEUAFSPBGEOVN7G7ABYQRUBF4DLINTCM33QIA, seg_pos={0 0}, upload=57077]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 139.59.144.22:42808->104.36.82.11:4115: use of closed network connection; write tcp 139.59.144.22:42808->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42808->104.36.82.11:4115: use of closed network connection Jul 26 23:07:14.244545905 (59h31m45.434844492s) [noise=true , address=104.36.82.15:4259 , piece_id=DOXQ4UVYFBSRTHQ2YR2WB6O4AYXNVZGFCCP6B6SBBUR3QGDAVLRQ, seg_pos={0 0}, upload=57077]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: read tcp 139.59.144.22:49378->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:14.244606299 (59h31m45.43490488s) [noise=true , address=104.36.82.4:4804 , piece_id=3OOX6YLC3AW5EGIH3YFQNAQWIRBFLBVBTZBRXT67YDT2SAC5T5XA, seg_pos={0 0}, upload=57077]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: read tcp 139.59.144.22:58724->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:14.244669396 (59h31m45.434967982s) [noise=true , address=104.36.82.12:4150 , piece_id=LEGJEJ5KJFOI7LZLJAPDUS6NOZZ4G37VWKHHUPJ7XEH5YLFDW2IQ, seg_pos={0 0}, upload=57077]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: read tcp 139.59.144.22:50646->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:14.244690100 (59h31m45.434988689s) [seg_pos={0 0}, upload=57077]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:07:14.353619397 (59h31m45.543917984s) [upload=57077]: Done uploading segments... Jul 26 23:07:14.353666753 (59h31m45.543965335s) [upload=57077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration:* 30.112770995s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.............................X..................X.....X] duration: 30.145113736s FAIL (2025-07-26 23:06:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.145141257s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X......X] duration: 30.11241244s FAIL (2025-07-26 23:06:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== GHtKeKio5fwRkmo5f8fms8a): protocol: rpc: dial tcp 77.90.31.13:4628: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:05:12.701565871 (59h29m43.89186444s) [seg_pos={0 0}, upload=57051]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 23:05:12.701669138 (59h29m43.891967724s) [upload=57051]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:05:12.701676335 (59h29m43.891974924s) [upload=57051]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:05:12.863518207 (59h29m44.053816786s) [upload=57051]: Done uploading segments... Jul 26 23:05:12.863544063 (59h29m44.053842645s) [upload=57051]: Upload finished. err: <nil> Jul 26 23:06:08.465337871 (59h30m39.655636459s) [upload=57067]: Starting upload Jul 26 23:06:08.465452738 (59h30m39.655751322s) [upload=57067]: Done starting upload Jul 26 23:06:08.465576893 (59h30m39.655875457s) [upload=57067]: Uploading segments... Jul 26 23:06:08.466065747 (59h30m39.656364337s) [upload=57067]: Got next segment. Inline: false Jul 26 23:06:08.466143855 (59h30m39.656442445s) [upload=57067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:06:08.466151565 (59h30m39.656450154s) [upload=57067]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:06:08.624828010 (59h30m39.8151266s) [seg_pos={0 0}, upload=57067]: Begin upload segment... Jul 26 23:06:08.624883950 (59h30m39.81518253s) [seg_pos={0 0}, upload=57067]: Done begin upload segment. Jul 26 23:06:08.624911032 (59h30m39.815209621s) [noise=true , address=45.91.116.4:4940 , piece_id=U7BXMTCED7PT2QLRXSXWA4IUY7DB3FEDU5DQJ4SU73NTO7TW3JSQ, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.625346404 (59h30m39.815644993s) [noise=true , address=104.36.82.9:4006 , piece_id=OCBZAQLDF7HMLEDURCFLVNUIFUARXKNXQA77C6MV4M46W4SAXDOA, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.625772131 (59h30m39.816070718s) [noise=true , address=45.91.116.7:4009 , piece_id=7OVUNWKQNQVUXUIBFZVHJT2KPOICA6NGUSRRPUQRZPETMUCYJI6Q, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.626236280 (59h30m39.816534869s) [noise=true , address=45.91.116.12:4225 , piece_id=2AOMGOD2FS223Q2EX7FTE3J4AVBF3YKQC45T44L3O6NBEWAUDXIA, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.626629653 (59h30m39.816928239s) [noise=true , address=45.91.116.14:4189 , piece_id=WTZQWH374NUISUPULL5DCZZDDKCQCP42IFMB2SMAW7HL4AFLVT3Q, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.627005141 (59h30m39.81730373s) [noise=true , address=77.90.31.16:4735 , piece_id=XQVUKI5J4WHI66H4WAKZF372DOE6X4T2IJ4BBYMZGZUAQ6HT7OXA, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.627375907 (59h30m39.817674495s) [noise=true , address=104.36.82.2:4730 , piece_id=KPZCBX5HPYSWRFVXSLZW4FTSKTIB2KJMXSOQEQQPN3BUNMW6TCXQ, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.627740359 (59h30m39.818038949s) [noise=true , address=104.36.82.3:4765 , piece_id=CCPDWKF6PFE4V23NKUROLPOWOOYAQYXPNQCCACSAVGZ35J7KQPDA, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.628073116 (59h30m39.818371706s) [noise=true , address=45.91.116.8:4045 , piece_id=L4NZBW5GDLVJMW3W6O36QKAFSWM536JG7YINYARAO2QFWCPMOP7Q, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.628448901 (59h30m39.818747492s) [noise=true , address=104.36.82.2:4731 , piece_id=YKTPIY6JK24COSN7MAKDW2KKXWDTUNOOTDYF5GRNT66Y26NWTVWA, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.628828758 (59h30m39.819127347s) [noise=true , address=104.36.82.8:4971 , piece_id=22RR4OFKFGHDYXTIZD4MNYO2QRZRWON6AF3N2MZMHVO2JU72Q7VA, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.629218235 (59h30m39.819516821s) [noise=true , address=77.90.31.14:4663 , piece_id=S3GICH4KSSQJ2NKQQYCOQVFABMVQUYRQIQXAKGIK6FW2UUNXWXNQ, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.629607587 (59h30m39.819906172s) [noise=true , address=45.91.116.6:4974 , piece_id=V4MZS6JR5UYVIL3SQFDGK5MEXQELD4BCLFGMGDJGC5IHMW672LGQ, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.630051430 (59h30m39.820350011s) [noise=true , address=104.36.82.4:4804 , piece_id=734GZ2MD7NNMRJVKXV2ZLAGZYHR3SOYRCIAEHGSHKRWZIA64VRLA, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.630424558 (59h30m39.82072315s) [noise=true , address=104.36.82.11:4113 , piece_id=MKXHTTUBHHSEGXFXLZ4H7G2FX4DG7SHQH52Y67C6SJHMKQA6TXXQ, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:08.631778018 (59h30m39.822076588s) [noise=true , address=77.90.31.12:4592 , piece_id=QJRVW3Q3OTY5I7SSVXCDOX3CR2EZNASU5JZVYRRKK6JS4OSHER4A, seg_pos={0 0}, upload=57067]: Uploading piece... Jul 26 23:06:09.254184006 (59h30m40.444482586s) [upload=57067]: Next returned nil segment Jul 26 23:06:09.254202997 (59h30m40.444501567s) [upload=57067]: Waiting for error group managing segments... Jul 26 23:06:09.403161029 (59h30m40.593459599s) [noise=true , address=45.91.116.4:4940 , piece_id=U7BXMTCED7PT2QLRXSXWA4IUY7DB3FEDU5DQJ4SU73NTO7TW3JSQ, seg_pos={0 0}, upload=57067]: Done uploading piece. err: <nil> Jul 26 23:06:09.408062127 (59h30m40.598360717s) [noise=true , address=45.91.116.6:4974 , piece_id=V4MZS6JR5UYVIL3SQFDGK5MEXQELD4BCLFGMGDJGC5IHMW672LGQ, seg_pos={0 0}, upload=57067]: Done uploading piece. err: <nil> Jul 26 23:06:09.413246158 (59h30m40.603544731s) [noise=true , address=77.90.31.14:4663 , piece_id=S3GICH4KSSQJ2NKQQYCOQVFABMVQUYRQIQXAKGIK6FW2UUNXWXNQ, seg_pos={0 0}, upload=57067]: Done uploading piece. err: <nil> Jul 26 23:06:09.414408956 (59h30m40.604707525s) [noise=true , address=45.91.116.7:4009 , piece_id=7OVUNWKQNQVUXUIBFZVHJT2KPOICA6NGUSRRPUQRZPETMUCYJI6Q, seg_pos={0 0}, upload=57067]: Done uploading piece. err: <nil> Jul 26 23:06:09.415069173 (59h30m40.605367758s) [noise=true , address=45.91.116.12:4225 , piece_id=2AOMGOD2FS223Q2EX7FTE3J4AVBF3YKQC45T44L3O6NBEWAUDXIA, seg_pos={0 0}, upload=57067]: Done uploading piece. err: <nil> Jul 26 23:06:09.423059535 (59h30m40.613358116s) [noise=true , address=45.91.116.14:4189 , piece_id=WTZQWH374NUISUPULL5DCZZDDKCQCP42IFMB2SMAW7HL4AFLVT3Q, seg_pos={0 0}, upload=57067]: Done uploading piece. err: <nil> Jul 26 23:06:09.431515601 (59h30m40.621814168s) [noise=true , address=45.91.116.8:4045 , piece_id=L4NZBW5GDLVJMW3W6O36QKAFSWM536JG7YINYARAO2QFWCPMOP7Q, seg_pos={0 0}, upload=57067]: Done uploading piece. err: <nil> Jul 26 23:06:09.915450187 (59h30m41.105748755s) [noise=true , address=104.36.82.8:4971 , piece_id=22RR4OFKFGHDYXTIZD4MNYO2QRZRWON6AF3N2MZMHVO2JU72Q7VA, seg_pos={0 0}, upload=57067]: Done uploading piece. err: <nil> Jul 26 23:06:10.787194284 (59h30m41.977492863s) [noise=true , address=77.90.31.12:4592 , piece_id=QJRVW3Q3OTY5I7SSVXCDOX3CR2EZNASU5JZVYRRKK6JS4OSHER4A, seg_pos={0 0}, upload=57067]: Done uploading piece. err: <nil> Jul 26 23:06:13.382776812 (59h30m44.573075385s) [noise=true , address=104.36.82.2:4731 , piece_id=YKTPIY6JK24COSN7MAKDW2KKXWDTUNOOTDYF5GRNT66Y26NWTVWA, seg_pos={0 0}, upload=57067]: Done uploading piece. err: <nil> Jul 26 23:06:13.493561558 (59h30m44.683860125s) [noise=true , address=104.36.82.2:4730 , piece_id=KPZCBX5HPYSWRFVXSLZW4FTSKTIB2KJMXSOQEQQPN3BUNMW6TCXQ, seg_pos={0 0}, upload=57067]: Done uploading piece. err: <nil> Jul 26 23:06:38.186607746 (59h31m9.376906325s) [noise=true , address=104.36.82.3:4765 , piece_id=CCPDWKF6PFE4V23NKUROLPOWOOYAQYXPNQCCACSAVGZ35J7KQPDA, seg_pos={0 0}, upload=57067]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:06:38.186625123 (59h31m9.376923702s) [noise=true , address=77.90.31.16:4735 , piece_id=XQVUKI5J4WHI66H4WAKZF372DOE6X4T2IJ4BBYMZGZUAQ6HT7OXA, seg_pos={0 0}, upload=57067]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:50260->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:06:38.186667437 (59h31m9.376966015s) [noise=true , address=104.36.82.9:4006 , piece_id=OCBZAQLDF7HMLEDURCFLVNUIFUARXKNXQA77C6MV4M46W4SAXDOA, seg_pos={0 0}, upload=57067]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 139.59.144.22:59824->104.36.82.9:4006: use of closed network connection; write tcp 139.59.144.22:59824->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59824->104.36.82.9:4006: use of closed network connection Jul 26 23:06:38.186715550 (59h31m9.37701412s) [noise=true , address=104.36.82.4:4804 , piece_id=734GZ2MD7NNMRJVKXV2ZLAGZYHR3SOYRCIAEHGSHKRWZIA64VRLA, seg_pos={0 0}, upload=57067]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:06:38.186765134 (59h31m9.377063713s) [noise=true , address=104.36.82.11:4113 , piece_id=MKXHTTUBHHSEGXFXLZ4H7G2FX4DG7SHQH52Y67C6SJHMKQA6TXXQ, seg_pos={0 0}, upload=57067]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:06:38.186781760 (59h31m9.37708034s) [seg_pos={0 0}, upload=57067]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:06:38.297398133 (59h31m9.487696716s) [upload=57067]: Done uploading segments... Jul 26 23:06:38.297427097 (59h31m9.487725671s) [upload=57067]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.11247301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [......XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.] duration: 9.731266682s FAIL (2025-07-26 23:06:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 23:05:18.708968998 (59h29m49.89926758s) [upload=57054]: Starting upload Jul 26 23:05:18.709044773 (59h29m49.899343358s) [upload=57054]: Done starting upload Jul 26 23:05:18.709220629 (59h29m49.899519205s) [upload=57054]: Uploading segments... Jul 26 23:05:18.709516005 (59h29m49.899814574s) [upload=57054]: Got next segment. Inline: false Jul 26 23:05:18.709583852 (59h29m49.899882421s) [upload=57054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:05:18.709590115 (59h29m49.899888685s) [upload=57054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:05:18.923399725 (59h29m50.113698311s) [seg_pos={0 0}, upload=57054]: Begin upload segment... Jul 26 23:05:18.923485955 (59h29m50.113784539s) [seg_pos={0 0}, upload=57054]: Done begin upload segment. Jul 26 23:05:18.923534778 (59h29m50.113833357s) [noise=true , address=77.90.31.17:4774 , piece_id=Z2AU3OPRLS2FD5NSDJCNWTJSOXOO2ZON5VOWSWPOE2DQSLXOWIKQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.924020835 (59h29m50.11431942s) [noise=true , address=77.90.31.11:4587 , piece_id=G5RTHKCSDQPB3TIJ7IQCYBYFQHZO7DBS6L7AVC4A6H5BCEE3L2RA, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.924098357 (59h29m50.114396936s) [noise=true , address=79.127.219.38:4001 , piece_id=2H6SEDLQ4TO4D55FIT4DXUXEPJDRQZWM63FGDIENUR6373NYGRNQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.924651940 (59h29m50.114950523s) [noise=true , address=77.90.31.19:4849 , piece_id=OQZLWQ24CHT5BNEPI5TVZFNZLKOWI53VQD2NT24SJG24JR4QQOPA, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.925051300 (59h29m50.115349883s) [noise=true , address=156.146.43.225:4449 , piece_id=ECMGHGO2JREBXIXV6V3LXMSTOUKNGU3XJAOBWGIHLEQI4KDG67VA, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.925355420 (59h29m50.115654001s) [noise=true , address=79.127.205.234:4615 , piece_id=6F5DUOOEIQUKEYP2WN2I47T23QHV2J7T4DDK377PTKFIFAJLFU5Q, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.925641160 (59h29m50.11593974s) [noise=true , address=79.127.205.233:4643 , piece_id=VGXLPA44PU3MEAH5BY7XGUBFUPNPPYBSOZFAF2TJFDFODVZASMFQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.925904075 (59h29m50.116202657s) [noise=true , address=79.127.205.234:4604 , piece_id=MPRJ44J6ODOAMR63UX46S7XOAARFVBOHEKS6VMVDRBC2T7H2BUBQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.926253380 (59h29m50.11655196s) [noise=true , address=45.91.116.14:4206 , piece_id=UTGXSRTVZ2IT2POB3CUPFKHINNNJJWBDVUMULPCPKHZN3RA4LPGA, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.926326460 (59h29m50.11662505s) [noise=true , address=77.90.31.11:4579 , piece_id=JFOHF7FOS6APM5TFIKMGYENQR2IJJJQPMUDENXQAETRXQ4USPT7Q, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.926638155 (59h29m50.116936725s) [noise=true , address=77.90.31.12:4605 , piece_id=ZIFSI4WGRBSK7BVUW64PYD6NAWXB4YUHBKSOMRPM5Z72OU7WTDVA, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.926674590 (59h29m50.116973178s) [noise=true , address=64.49.13.138:4482 , piece_id=BUXNLQ6UERAZ5X5IQPEQML3TA6SWHOPPV5QGSKERY22NYNSU467A, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.927000311 (59h29m50.117298901s) [noise=true , address=79.127.205.235:4680 , piece_id=2LEW3WNIUCBGRWZEEAMNRBFIHOMRREWVJ3I72KTTOUHW5UCIIRKQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.927030056 (59h29m50.117328646s) [noise=true , address=79.127.201.214:4321 , piece_id=KS463ANUNVWE5UYLP5OPHRIYXZPYX4URJFIW4W6ECS2FHY5JHSGA, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.927358805 (59h29m50.117657396s) [noise=true , address=79.127.219.47:4048 , piece_id=H33CNC5B7JKXZ7VXVGWUBNBO6QEGHNXLUNR5X4REMFXW6PIVM6PQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.927384346 (59h29m50.117682937s) [noise=true , address=79.127.205.245:4240 , piece_id=OLXAXV6GXWJZ5TVLXK7M5HVWQGA2IKPWK5BBFPUQNVKRO6TGXYQQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.927768570 (59h29m50.118067155s) [noise=true , address=79.127.205.233:4623 , piece_id=UJREZUHFZFWZAEGFC6PRIQYP2RGXG7K27AGSAZ47IW3LG6S3TKJQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.928096908 (59h29m50.118395492s) [noise=true , address=77.90.31.19:4847 , piece_id=BZ7QL5ETNLSOFODZGLMR2IOZGNGGKOJ7NBA6G6KVO52IFFEXQZSQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:19.523370422 (59h29m50.713669003s) [upload=57054]: Next returned nil segment Jul 26 23:05:19.523389297 (59h29m50.713687867s) [upload=57054]: Waiting for error group managing segments... Jul 26 23:05:20.044169079 (59h29m51.234467662s) [noise=true , address=79.127.219.38:4001 , piece_id=2H6SEDLQ4TO4D55FIT4DXUXEPJDRQZWM63FGDIENUR6373NYGRNQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.206206826 (59h29m51.396505405s) [noise=true , address=79.127.205.235:4680 , piece_id=2LEW3WNIUCBGRWZEEAMNRBFIHOMRREWVJ3I72KTTOUHW5UCIIRKQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.253754198 (59h29m51.444052773s) [noise=true , address=79.127.205.245:4240 , piece_id=OLXAXV6GXWJZ5TVLXK7M5HVWQGA2IKPWK5BBFPUQNVKRO6TGXYQQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.254626805 (59h29m51.444925376s) [noise=true , address=79.127.201.214:4321 , piece_id=KS463ANUNVWE5UYLP5OPHRIYXZPYX4URJFIW4W6ECS2FHY5JHSGA, seg_pos={0 0}, upload=57054]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:20.255956426 (59h29m51.446254992s) [noise=true , address=79.127.205.234:4604 , piece_id=MPRJ44J6ODOAMR63UX46S7XOAARFVBOHEKS6VMVDRBC2T7H2BUBQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.290443588 (59h29m51.480742167s) [noise=true , address=79.127.205.233:4623 , piece_id=UJREZUHFZFWZAEGFC6PRIQYP2RGXG7K27AGSAZ47IW3LG6S3TKJQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.293652344 (59h29m51.483950918s) [noise=true , address=79.127.205.234:4615 , piece_id=6F5DUOOEIQUKEYP2WN2I47T23QHV2J7T4DDK377PTKFIFAJLFU5Q, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.327159474 (59h29m51.517458039s) [noise=true , address=79.127.205.233:4643 , piece_id=VGXLPA44PU3MEAH5BY7XGUBFUPNPPYBSOZFAF2TJFDFODVZASMFQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.488734630 (59h29m51.679033209s) [noise=true , address=77.90.31.17:4774 , piece_id=Z2AU3OPRLS2FD5NSDJCNWTJSOXOO2ZON5VOWSWPOE2DQSLXOWIKQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:21.807897818 (59h29m52.998196399s) [noise=true , address=64.49.13.138:4482 , piece_id=BUXNLQ6UERAZ5X5IQPEQML3TA6SWHOPPV5QGSKERY22NYNSU467A, seg_pos={0 0}, upload=57054]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:23.790249760 (59h29m54.980548333s) [noise=true , address=156.146.43.225:4449 , piece_id=ECMGHGO2JREBXIXV6V3LXMSTOUKNGU3XJAOBWGIHLEQI4KDG67VA, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:24.181600144 (59h29m55.37189871s) [noise=true , address=79.127.219.47:4048 , piece_id=H33CNC5B7JKXZ7VXVGWUBNBO6QEGHNXLUNR5X4REMFXW6PIVM6PQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:25.694689962 (59h29m56.884988541s) [noise=true , address=77.90.31.12:4605 , piece_id=ZIFSI4WGRBSK7BVUW64PYD6NAWXB4YUHBKSOMRPM5Z72OU7WTDVA, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:33.455007321 (59h30m4.645305891s) [noise=true , address=45.91.116.14:4206 , piece_id=UTGXSRTVZ2IT2POB3CUPFKHINNNJJWBDVUMULPCPKHZN3RA4LPGA, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:59.978364109 (59h30m31.168662695s) [noise=true , address=77.90.31.19:4847 , piece_id=BZ7QL5ETNLSOFODZGLMR2IOZGNGGKOJ7NBA6G6KVO52IFFEXQZSQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:06:18.433659087 (59h30m49.623957657s) [noise=true , address=77.90.31.19:4849 , piece_id=OQZLWQ24CHT5BNEPI5TVZFNZLKOWI53VQD2NT24SJG24JR4QQOPA, seg_pos={0 0}, upload=57054]: Done uploading piece. err: ecclient: upload failed (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, address:77.90.31.19:4849): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:06:18.433786536 (59h30m49.624085116s) [noise=true , address=77.90.31.11:4579 , piece_id=JFOHF7FOS6APM5TFIKMGYENQR2IJJJQPMUDENXQAETRXQ4USPT7Q, seg_pos={0 0}, upload=57054]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 139.59.144.22:44938->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:06:18.433889525 (59h30m49.624188092s) [noise=true , address=77.90.31.11:4587 , piece_id=G5RTHKCSDQPB3TIJ7IQCYBYFQHZO7DBS6L7AVC4A6H5BCEE3L2RA, seg_pos={0 0}, upload=57054]: Done uploading piece. err: ecclient: upload failed (node:1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, address:77.90.31.11:4587): protocol: rpc: dial tcp 77.90.31.11:4587: i/o timeout; conn closed: op Jul 26 23:06:18.433967149 (59h30m49.624265733s) [seg_pos={0 0}, upload=57054]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:06:18.557224847 (59h30m49.74752342s) [upload=57054]: Done uploading segments... Jul 26 23:06:18.557251720 (59h30m49.74755029s) [upload=57054]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.731315228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [.......XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX] duration: 1m0.125411786s FAIL (2025-07-26 23:06:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 23:05:18.708968998 (59h29m49.89926758s) [upload=57054]: Starting upload Jul 26 23:05:18.709044773 (59h29m49.899343358s) [upload=57054]: Done starting upload Jul 26 23:05:18.709220629 (59h29m49.899519205s) [upload=57054]: Uploading segments... Jul 26 23:05:18.709516005 (59h29m49.899814574s) [upload=57054]: Got next segment. Inline: false Jul 26 23:05:18.709583852 (59h29m49.899882421s) [upload=57054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:05:18.709590115 (59h29m49.899888685s) [upload=57054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:05:18.923399725 (59h29m50.113698311s) [seg_pos={0 0}, upload=57054]: Begin upload segment... Jul 26 23:05:18.923485955 (59h29m50.113784539s) [seg_pos={0 0}, upload=57054]: Done begin upload segment. Jul 26 23:05:18.923534778 (59h29m50.113833357s) [noise=true , address=77.90.31.17:4774 , piece_id=Z2AU3OPRLS2FD5NSDJCNWTJSOXOO2ZON5VOWSWPOE2DQSLXOWIKQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.924020835 (59h29m50.11431942s) [noise=true , address=77.90.31.11:4587 , piece_id=G5RTHKCSDQPB3TIJ7IQCYBYFQHZO7DBS6L7AVC4A6H5BCEE3L2RA, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.924098357 (59h29m50.114396936s) [noise=true , address=79.127.219.38:4001 , piece_id=2H6SEDLQ4TO4D55FIT4DXUXEPJDRQZWM63FGDIENUR6373NYGRNQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.924651940 (59h29m50.114950523s) [noise=true , address=77.90.31.19:4849 , piece_id=OQZLWQ24CHT5BNEPI5TVZFNZLKOWI53VQD2NT24SJG24JR4QQOPA, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.925051300 (59h29m50.115349883s) [noise=true , address=156.146.43.225:4449 , piece_id=ECMGHGO2JREBXIXV6V3LXMSTOUKNGU3XJAOBWGIHLEQI4KDG67VA, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.925355420 (59h29m50.115654001s) [noise=true , address=79.127.205.234:4615 , piece_id=6F5DUOOEIQUKEYP2WN2I47T23QHV2J7T4DDK377PTKFIFAJLFU5Q, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.925641160 (59h29m50.11593974s) [noise=true , address=79.127.205.233:4643 , piece_id=VGXLPA44PU3MEAH5BY7XGUBFUPNPPYBSOZFAF2TJFDFODVZASMFQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.925904075 (59h29m50.116202657s) [noise=true , address=79.127.205.234:4604 , piece_id=MPRJ44J6ODOAMR63UX46S7XOAARFVBOHEKS6VMVDRBC2T7H2BUBQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.926253380 (59h29m50.11655196s) [noise=true , address=45.91.116.14:4206 , piece_id=UTGXSRTVZ2IT2POB3CUPFKHINNNJJWBDVUMULPCPKHZN3RA4LPGA, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.926326460 (59h29m50.11662505s) [noise=true , address=77.90.31.11:4579 , piece_id=JFOHF7FOS6APM5TFIKMGYENQR2IJJJQPMUDENXQAETRXQ4USPT7Q, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.926638155 (59h29m50.116936725s) [noise=true , address=77.90.31.12:4605 , piece_id=ZIFSI4WGRBSK7BVUW64PYD6NAWXB4YUHBKSOMRPM5Z72OU7WTDVA, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.926674590 (59h29m50.116973178s) [noise=true , address=64.49.13.138:4482 , piece_id=BUXNLQ6UERAZ5X5IQPEQML3TA6SWHOPPV5QGSKERY22NYNSU467A, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.927000311 (59h29m50.117298901s) [noise=true , address=79.127.205.235:4680 , piece_id=2LEW3WNIUCBGRWZEEAMNRBFIHOMRREWVJ3I72KTTOUHW5UCIIRKQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.927030056 (59h29m50.117328646s) [noise=true , address=79.127.201.214:4321 , piece_id=KS463ANUNVWE5UYLP5OPHRIYXZPYX4URJFIW4W6ECS2FHY5JHSGA, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.927358805 (59h29m50.117657396s) [noise=true , address=79.127.219.47:4048 , piece_id=H33CNC5B7JKXZ7VXVGWUBNBO6QEGHNXLUNR5X4REMFXW6PIVM6PQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.927384346 (59h29m50.117682937s) [noise=true , address=79.127.205.245:4240 , piece_id=OLXAXV6GXWJZ5TVLXK7M5HVWQGA2IKPWK5BBFPUQNVKRO6TGXYQQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.927768570 (59h29m50.118067155s) [noise=true , address=79.127.205.233:4623 , piece_id=UJREZUHFZFWZAEGFC6PRIQYP2RGXG7K27AGSAZ47IW3LG6S3TKJQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:18.928096908 (59h29m50.118395492s) [noise=true , address=77.90.31.19:4847 , piece_id=BZ7QL5ETNLSOFODZGLMR2IOZGNGGKOJ7NBA6G6KVO52IFFEXQZSQ, seg_pos={0 0}, upload=57054]: Uploading piece... Jul 26 23:05:19.523370422 (59h29m50.713669003s) [upload=57054]: Next returned nil segment Jul 26 23:05:19.523389297 (59h29m50.713687867s) [upload=57054]: Waiting for error group managing segments... Jul 26 23:05:20.044169079 (59h29m51.234467662s) [noise=true , address=79.127.219.38:4001 , piece_id=2H6SEDLQ4TO4D55FIT4DXUXEPJDRQZWM63FGDIENUR6373NYGRNQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.206206826 (59h29m51.396505405s) [noise=true , address=79.127.205.235:4680 , piece_id=2LEW3WNIUCBGRWZEEAMNRBFIHOMRREWVJ3I72KTTOUHW5UCIIRKQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.253754198 (59h29m51.444052773s) [noise=true , address=79.127.205.245:4240 , piece_id=OLXAXV6GXWJZ5TVLXK7M5HVWQGA2IKPWK5BBFPUQNVKRO6TGXYQQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.254626805 (59h29m51.444925376s) [noise=true , address=79.127.201.214:4321 , piece_id=KS463ANUNVWE5UYLP5OPHRIYXZPYX4URJFIW4W6ECS2FHY5JHSGA, seg_pos={0 0}, upload=57054]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:20.255956426 (59h29m51.446254992s) [noise=true , address=79.127.205.234:4604 , piece_id=MPRJ44J6ODOAMR63UX46S7XOAARFVBOHEKS6VMVDRBC2T7H2BUBQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.290443588 (59h29m51.480742167s) [noise=true , address=79.127.205.233:4623 , piece_id=UJREZUHFZFWZAEGFC6PRIQYP2RGXG7K27AGSAZ47IW3LG6S3TKJQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.293652344 (59h29m51.483950918s) [noise=true , address=79.127.205.234:4615 , piece_id=6F5DUOOEIQUKEYP2WN2I47T23QHV2J7T4DDK377PTKFIFAJLFU5Q, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.327159474 (59h29m51.517458039s) [noise=true , address=79.127.205.233:4643 , piece_id=VGXLPA44PU3MEAH5BY7XGUBFUPNPPYBSOZFAF2TJFDFODVZASMFQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:20.488734630 (59h29m51.679033209s) [noise=true , address=77.90.31.17:4774 , piece_id=Z2AU3OPRLS2FD5NSDJCNWTJSOXOO2ZON5VOWSWPOE2DQSLXOWIKQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:21.807897818 (59h29m52.998196399s) [noise=true , address=64.49.13.138:4482 , piece_id=BUXNLQ6UERAZ5X5IQPEQML3TA6SWHOPPV5QGSKERY22NYNSU467A, seg_pos={0 0}, upload=57054]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:23.790249760 (59h29m54.980548333s) [noise=true , address=156.146.43.225:4449 , piece_id=ECMGHGO2JREBXIXV6V3LXMSTOUKNGU3XJAOBWGIHLEQI4KDG67VA, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:24.181600144 (59h29m55.37189871s) [noise=true , address=79.127.219.47:4048 , piece_id=H33CNC5B7JKXZ7VXVGWUBNBO6QEGHNXLUNR5X4REMFXW6PIVM6PQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:25.694689962 (59h29m56.884988541s) [noise=true , address=77.90.31.12:4605 , piece_id=ZIFSI4WGRBSK7BVUW64PYD6NAWXB4YUHBKSOMRPM5Z72OU7WTDVA, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:33.455007321 (59h30m4.645305891s) [noise=true , address=45.91.116.14:4206 , piece_id=UTGXSRTVZ2IT2POB3CUPFKHINNNJJWBDVUMULPCPKHZN3RA4LPGA, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:05:59.978364109 (59h30m31.168662695s) [noise=true , address=77.90.31.19:4847 , piece_id=BZ7QL5ETNLSOFODZGLMR2IOZGNGGKOJ7NBA6G6KVO52IFFEXQZSQ, seg_pos={0 0}, upload=57054]: Done uploading piece. err: <nil> Jul 26 23:06:18.433659087 (59h30m49.623957657s) [noise=true , address=77.90.31.19:4849 , piece_id=OQZLWQ24CHT5BNEPI5TVZFNZLKOWI53VQD2NT24SJG24JR4QQOPA, seg_pos={0 0}, upload=57054]: Done uploading piece. err: ecclient: upload failed (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, address:77.90.31.19:4849): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:06:18.433786536 (59h30m49.624085116s) [noise=true , address=77.90.31.11:4579 , piece_id=JFOHF7FOS6APM5TFIKMGYENQR2IJJJQPMUDENXQAETRXQ4USPT7Q, seg_pos={0 0}, upload=57054]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 139.59.144.22:44938->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:06:18.433889525 (59h30m49.624188092s) [noise=true , address=77.90.31.11:4587 , piece_id=G5RTHKCSDQPB3TIJ7IQCYBYFQHZO7DBS6L7AVC4A6H5BCEE3L2RA, seg_pos={0 0}, upload=57054]: Done uploading piece. err: ecclient: upload failed (node:1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, address:77.90.31.11:4587): protocol: rpc: dial tcp 77.90.31.11:4587: i/o timeout; conn closed: op Jul 26 23:06:18.433967149 (59h30m49.624265733s) [seg_pos={0 0}, upload=57054]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:06:18.557224847 (59h30m49.74752342s) [upload=57054]: Done uploading segments... Jul 26 23:06:18.557251720 (59h30m49.74755029s) [upload=57054]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.125434147s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [......X..X...X.XX...X..X.XX......X.X.......X.......X........] duration: 16.562025214s FAIL (2025-07-26 23:06:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.562047807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.......X..X...X.XX...X..X.XX......X.X.......X.......X.......] duration: 24.391404751s FAIL (2025-07-26 23:06:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.391424442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.............................X..................X.....] duration: 2.77647709s FAIL (2025-07-26 23:05:48 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.776528441s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X.......X..X...X.XX...X..X.XX......X.X.......X.......X......] duration: 17.551259812s FAIL (2025-07-26 23:05:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.551314378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.............................X..................X....] duration: 2.381905673s FAIL (2025-07-26 23:05:26 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.381933815s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [........XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXXX] duration: 1m0.123970477s FAIL (2025-07-26 23:05:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload Jul 26 23:04:18.439684601 (59h28m49.62998317s) [upload=57040]: Uploading segments... Jul 26 23:04:18.439737611 (59h28m49.630036198s) [upload=57040]: Got next segment. Inline: false Jul 26 23:04:18.439768977 (59h28m49.630067555s) [upload=57040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:04:18.439774356 (59h28m49.630072935s) [upload=57040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:04:18.698751073 (59h28m49.889049659s) [seg_pos={0 0}, upload=57040]: Begin upload segment... Jul 26 23:04:18.698841876 (59h28m49.889140441s) [seg_pos={0 0}, upload=57040]: Done begin upload segment. Jul 26 23:04:18.698871902 (59h28m49.889170484s) [noise=true , address=77.90.31.15:4732 , piece_id=V755XFXPCB45RFMMHFIDGAD7QFWFR4KT4Y7PQPU5EH7ORW2BEOUA, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.698887686 (59h28m49.889186302s) [noise=true , address=77.90.31.19:4850 , piece_id=T72HR53I7IFKWDSMJUO4ENCTACKB43G3VHFPKVIFXLH2IHXZULVA, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.699396754 (59h28m49.889695629s) [noise=true , address=79.127.205.235:4670 , piece_id=EREXGMLP5DB6PUOXHBPWR4RCTJTYHMCNOT7M2VOKGIHTDLROJFQA, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.699712563 (59h28m49.890011142s) [noise=true , address=79.127.205.245:4246 , piece_id=OQFAURGXGBZ7KYQ57O22MEZZMO3QPAIG25PF5DM64WMOWYGBPEDA, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.699968933 (59h28m49.890267522s) [noise=true , address=77.90.31.21:4888 , piece_id=WWROAFAT5QU4EFUX7ISYAINBAQZWKVW5RKXVKBJU4SGHNMVFHTWQ, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.699990238 (59h28m49.890288817s) [noise=true , address=79.127.205.234:4619 , piece_id=OLUISM3NVVGRCCBOKOK72LQUAIHQVTCWSHAYVZQJK5JGE6EJLTVA, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.700360795 (59h28m49.890659374s) [noise=true , address=79.127.201.215:4353 , piece_id=DEFS4J2JY5AS43DW3ZYIJP32K77O5QUJQN7DET26JCUOXBCB42TQ, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.700918280 (59h28m49.891216859s) [noise=true , address=77.90.31.21:4887 , piece_id=X52XIQ2DZNDGSX7PMRMU7UIM2QG7UB6VVLI6T274K6OFYXFBQ2DA, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.701221970 (59h28m49.891520553s) [noise=true , address=79.127.219.38:4009 , piece_id=LSZEQ72UGTVTEL6SA3GPZJZD5SDOHJSCOP2VSZ6TAKKJOKQZEN4Q, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.701532651 (59h28m49.891831236s) [noise=true , address=79.127.205.245:4247 , piece_id=2GNWA3WFBBWCR4R52VKA2CEY7AKVHGCUQLSZBDMORJG3YKTSYURQ, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.701801851 (59h28m49.892100433s) [noise=true , address=79.127.205.246:4282 , piece_id=NBI2RHIGX4YZHWCIWAWLVYGZQH3FZVHOW75FOUV2ZUXDJ2I6ZOWA, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.701991820 (59h28m49.892290402s) [noise=true , address=79.127.219.48:4090 , piece_id=TKEHZUVGHTH5AEURTMKDK3U4EJU7AARK4XJ2K2RGG7TULWL4CSZQ, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.702424203 (59h28m49.892722788s) [noise=true , address=79.127.219.51:4212 , piece_id=XMTGZQIXRBQP4QAATMPABQAUK35Q3R4ZON3EXLJDWQN2ZOFAZXLA, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.702491675 (59h28m49.892790253s) [noise=true , address=79.127.201.215:4352 , piece_id=Q55EMJM23GOTSRK5UNDH3EIXKQCNKKKSN55DFKWHUGTKXURU32SA, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.702806069 (59h28m49.893104654s) [noise=true , address=77.90.31.11:4574 , piece_id=FUXWS64BITLDA3H6MAFA2MTQQXYTBSJO5SXMAE4AFNYD5XQVEVJA, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.703094583 (59h28m49.893393163s) [noise=true , address=45.91.116.5:4310 , piece_id=L3KFPKPPIYQGKJA2VIOMAN273SMH54H7SKQ3L2L4IRTLUQCU2BPQ, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.703248148 (59h28m49.893546737s) [noise=true , address=79.127.219.51:4210 , piece_id=NLZGEGA6BZEPIUIVHINKDL6CPZMF4JHPFE6GCURGPWMUSCRHQ66A, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:18.703668522 (59h28m49.893967102s) [noise=true , address=79.127.201.215:4351 , piece_id=D5O2SKHNUNQG3OWPCC4TQZUG67MNWMDKKPGZWNREWDL73H4BKXYA, seg_pos={0 0}, upload=57040]: Uploading piece... Jul 26 23:04:19.104407063 (59h28m50.294705643s) [noise=true , address=79.127.219.51:4210 , piece_id=NLZGEGA6BZEPIUIVHINKDL6CPZMF4JHPFE6GCURGPWMUSCRHQ66A, seg_pos={0 0}, upload=57040]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:19.303898698 (59h28m50.494197264s) [upload=57040]: Next returned nil segment Jul 26 23:04:19.303915952 (59h28m50.494214517s) [upload=57040]: Waiting for error group managing segments... Jul 26 23:04:19.673808520 (59h28m50.8641071s) [noise=true , address=79.127.219.38:4009 , piece_id=LSZEQ72UGTVTEL6SA3GPZJZD5SDOHJSCOP2VSZ6TAKKJOKQZEN4Q, seg_pos={0 0}, upload=57040]: Done uploading piece. err: <nil> Jul 26 23:04:19.974306248 (59h28m51.164604817s) [noise=true , address=79.127.205.246:4282 , piece_id=NBI2RHIGX4YZHWCIWAWLVYGZQH3FZVHOW75FOUV2ZUXDJ2I6ZOWA, seg_pos={0 0}, upload=57040]: Done uploading piece. err: <nil> Jul 26 23:04:20.010740445 (59h28m51.201039025s) [noise=true , address=79.127.205.245:4247 , piece_id=2GNWA3WFBBWCR4R52VKA2CEY7AKVHGCUQLSZBDMORJG3YKTSYURQ, seg_pos={0 0}, upload=57040]: Done uploading piece. err: <nil> Jul 26 23:04:20.023290874 (59h28m51.213589458s) [noise=true , address=79.127.205.235:4670 , piece_id=EREXGMLP5DB6PUOXHBPWR4RCTJTYHMCNOT7M2VOKGIHTDLROJFQA, seg_pos={0 0}, upload=57040]: Done uploading piece. err: <nil> Jul 26 23:04:20.029296443 (59h28m51.219595012s) [noise=true , address=79.127.205.245:4246 , piece_id=OQFAURGXGBZ7KYQ57O22MEZZMO3QPAIG25PF5DM64WMOWYGBPEDA, seg_pos={0 0}, upload=57040]: Done uploading piece. err: <nil> Jul 26 23:04:20.062989665 (59h28m51.253288231s) [noise=true , address=79.127.205.234:4619 , piece_id=OLUISM3NVVGRCCBOKOK72LQUAIHQVTCWSHAYVZQJK5JGE6EJLTVA, seg_pos={0 0}, upload=57040]: Done uploading piece. err: <nil> Jul 26 23:04:21.324034619 (59h28m52.514333207s) [noise=true , address=79.127.219.51:4212 , piece_id=XMTGZQIXRBQP4QAATMPABQAUK35Q3R4ZON3EXLJDWQN2ZOFAZXLA, seg_pos={0 0}, upload=57040]: Done uploading piece. err: <nil> Jul 26 23:04:23.159535873 (59h28m54.349834458s) [noise=true , address=79.127.219.48:4090 , piece_id=TKEHZUVGHTH5AEURTMKDK3U4EJU7AARK4XJ2K2RGG7TULWL4CSZQ, seg_pos={0 0}, upload=57040]: Done uploading piece. err: <nil> Jul 26 23:04:46.318964155 (59h29m17.509262725s) [noise=true , address=45.91.116.5:4310 , piece_id=L3KFPKPPIYQGKJA2VIOMAN273SMH54H7SKQ3L2L4IRTLUQCU2BPQ, seg_pos={0 0}, upload=57040]: Done uploading piece. err: <nil> Jul 26 23:05:02.355941769 (59h29m33.546240355s) [noise=true , address=79.127.201.215:4353 , piece_id=DEFS4J2JY5AS43DW3ZYIJP32K77O5QUJQN7DET26JCUOXBCB42TQ, seg_pos={0 0}, upload=57040]: Done uploading piece. err: <nil> Jul 26 23:05:08.531430486 (59h29m39.72172906s) [noise=true , address=79.127.201.215:4351 , piece_id=D5O2SKHNUNQG3OWPCC4TQZUG67MNWMDKKPGZWNREWDL73H4BKXYA, seg_pos={0 0}, upload=57040]: Done uploading piece. err: <nil> Jul 26 23:05:11.722168088 (59h29m42.912466671s) [noise=true , address=79.127.201.215:4352 , piece_id=Q55EMJM23GOTSRK5UNDH3EIXKQCNKKKSN55DFKWHUGTKXURU32SA, seg_pos={0 0}, upload=57040]: Done uploading piece. err: <nil> Jul 26 23:05:16.166672130 (59h29m47.356970696s) [noise=true , address=77.90.31.19:4850 , piece_id=T72HR53I7IFKWDSMJUO4ENCTACKB43G3VHFPKVIFXLH2IHXZULVA, seg_pos={0 0}, upload=57040]: Done uploading piece. err: <nil> Jul 26 23:05:18.162492169 (59h29m49.352790751s) [noise=true , address=77.90.31.21:4887 , piece_id=X52XIQ2DZNDGSX7PMRMU7UIM2QG7UB6VVLI6T274K6OFYXFBQ2DA, seg_pos={0 0}, upload=57040]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: write tcp 139.59.144.22:51870->77.90.31.21:4887: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:05:18.162513405 (59h29m49.352811971s) [noise=true , address=77.90.31.11:4574 , piece_id=FUXWS64BITLDA3H6MAFA2MTQQXYTBSJO5SXMAE4AFNYD5XQVEVJA, seg_pos={0 0}, upload=57040]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 139.59.144.22:52156->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:05:18.162574382 (59h29m49.352872965s) [noise=true , address=77.90.31.15:4732 , piece_id=V755XFXPCB45RFMMHFIDGAD7QFWFR4KT4Y7PQPU5EH7ORW2BEOUA, seg_pos={0 0}, upload=57040]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: write tcp 139.59.144.22:34052->77.90.31.15:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:05:18.162738779 (59h29m49.353037363s) [noise=true , address=77.90.31.21:4888 , piece_id=WWROAFAT5QU4EFUX7ISYAINBAQZWKVW5RKXVKBJU4SGHNMVFHTWQ, seg_pos={0 0}, upload=57040]: Done uploading piece. err: ecclient: upload failed (node:12BiqwcDuvobZabx3jxDhXf27Ei8oLFS6wLp3qbpZCtJ4ZhQEZA, address:77.90.31.21:4888): protocol: read tcp 139.59.144.22:45580->77.90.31.21:4888: use of closed network connection; context deadline exceeded Jul 26 23:05:18.162771590 (59h29m49.353070179s) [seg_pos={0 0}, upload=57040]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:05:18.284578992 (59h29m49.474877565s) [upload=57040]: Done uploading segments... Jul 26 23:05:18.284610208 (59h29m49.474908791s) [upload=57040]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.123994072s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X.......X..X...X.XX...X..X.XX......X.X.......X.......X.....] duration: 34.045965126s FAIL (2025-07-26 23:05:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.045988853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.47158265s FAIL (2025-07-26 23:05:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 280->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:44.058547940 (59h24m15.24884652s) [seg_pos={0 0}, upload=56969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:59:44.058641678 (59h24m15.248940257s) [upload=56969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:59:44.058646789 (59h24m15.248945359s) [upload=56969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:59:44.224851647 (59h24m15.415150235s) [upload=56969]: Done uploading segments... Jul 26 22:59:44.224876852 (59h24m15.415175436s) [upload=56969]: Upload finished. err: <nil> Jul 26 23:00:00.341276866 (59h24m31.531575453s) [upload=56972]: Starting upload Jul 26 23:00:00.341377270 (59h24m31.531675865s) [upload=56972]: Done starting upload Jul 26 23:00:00.341505637 (59h24m31.531804221s) [upload=56972]: Uploading segments... Jul 26 23:00:00.341940351 (59h24m31.532238944s) [upload=56972]: Got next segment. Inline: false Jul 26 23:00:00.342010661 (59h24m31.532309252s) [upload=56972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:00:00.342017759 (59h24m31.532316348s) [upload=56972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:00:00.455411486 (59h24m31.64571009s) [seg_pos={0 0}, upload=56972]: Begin upload segment... Jul 26 23:00:00.455493514 (59h24m31.645792104s) [seg_pos={0 0}, upload=56972]: Done begin upload segment. Jul 26 23:00:00.455540024 (59h24m31.645838613s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.455561207 (59h24m31.645859795s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456074349 (59h24m31.646372943s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456243971 (59h24m31.64654256s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456605120 (59h24m31.646903743s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456973436 (59h24m31.647272026s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457050035 (59h24m31.647348625s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457398570 (59h24m31.64769716s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457455587 (59h24m31.647754178s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457779166 (59h24m31.648077758s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458115180 (59h24m31.648413769s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458194315 (59h24m31.648492905s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458592315 (59h24m31.648890905s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458651549 (59h24m31.648950139s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.459145371 (59h24m31.649443962s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.462726202 (59h24m31.653024793s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:01.079541782 (59h24m32.269840354s) [upload=56972]: Next returned nil segment Jul 26 23:00:01.079566166 (59h24m32.269864784s) [upload=56972]: Waiting for error group managing segments... Jul 26 23:00:01.210918643 (59h24m32.401217229s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.228724770 (59h24m32.419023355s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232110801 (59h24m32.422409369s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232305605 (59h24m32.422604172s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234390772 (59h24m32.424689363s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234614195 (59h24m32.424912769s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246294708 (59h24m32.43659329s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246456357 (59h24m32.43675493s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.247086738 (59h24m32.437385324s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.248804736 (59h24m32.43910332s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.302224542 (59h24m32.492523129s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:30.042884997 (59h25m1.233183578s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:48592->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043116623 (59h25m1.233415194s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043228739 (59h25m1.233527306s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043315635 (59h25m1.233614206s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043325410 (59h25m1.233623995s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:50064->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043373737 (59h25m1.23367233s) [seg_pos={0 0}, upload=56972]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:00:30.159169528 (59h25m1.349468101s) [upload=56972]: Done uploading segments... Jul 26 23:00:30.159218327 (59h25m1.34951691s) [upload=56972]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.471618553s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)