Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
8.092989ms remote:
"tcp:185.244.226.3:443" response headers:
Content-Type=application/xml X-Xss-Protection=1; mode=block Date=Thu, 26 Feb 2026 18:08:25 GMT Accept-Ranges=bytes Server=Storj Vary=Origin X-Amz-Request-Id=1897DE36CB19BFB6 X-Request-Id=D9H9uh6Dzoz Content-Length=274 Content-Security-Policy=block-all-mixed-content target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/7d43beda67176c28
FAIL (2026-02-26 18:08:25 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE36CB19BFB6</RequestId><HostId></HostId></Error>"
Duration: 8.112978ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
2.390044215s trace:
https://jaeger.dev.storj.io/trace/68a06934c0a05f04
FAIL (2026-02-26 18:08:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
eb 26 18:02:40.761285883 (213h45m45.695740025s) [upload=188070]: Starting upload
Feb 26 18:02:40.761395708 (213h45m45.695849854s) [upload=188070]: Done starting upload
Feb 26 18:02:40.761697240 (213h45m45.696151374s) [upload=188070]: Uploading segments...
Feb 26 18:02:40.762148293 (213h45m45.696602435s) [upload=188070]: Got next segment. Inline: false
Feb 26 18:02:40.762222848 (213h45m45.696676987s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 26 18:02:40.762230722 (213h45m45.69668487s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 26 18:02:41.270627701 (213h45m46.20508184s) [seg_pos={0 0}, upload=188070]: Begin upload segment...
Feb 26 18:02:41.270720334 (213h45m46.205174474s) [seg_pos={0 0}, upload=188070]: Done begin upload segment.
Feb 26 18:02:41.270802481 (213h45m46.205256658s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272042027 (213h45m46.206496159s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272283707 (213h45m46.20673785s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272744753 (213h45m46.207198908s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272938814 (213h45m46.207392958s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273277379 (213h45m46.207731498s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273552846 (213h45m46.208006989s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273597171 (213h45m46.208051303s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274079549 (213h45m46.208533671s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274139867 (213h45m46.208594009s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274541012 (213h45m46.208995213s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274823659 (213h45m46.209277801s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.275440186 (213h45m46.209894339s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.276035884 (213h45m46.210490026s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.277403363 (213h45m46.211857506s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.279624637 (213h45m46.214078779s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.889995128 (213h45m46.824449265s) [upload=188070]: Next returned nil segment
Feb 26 18:02:41.890012492 (213h45m46.824466626s) [upload=188070]: Waiting for error group managing segments...
Feb 26 18:02:42.025533148 (213h45m46.959987291s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.026652488 (213h45m46.961106618s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.067448882 (213h45m47.001903023s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.074637015 (213h45m47.009091156s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085003774 (213h45m47.019457898s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085343706 (213h45m47.019797831s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.129629735 (213h45m47.064083879s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.152804779 (213h45m47.087258916s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:55.018263081 (213h45m59.952717215s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:56.602166399 (213h46m1.536620544s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:03.112945430 (213h46m8.047399567s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:08.984934397 (213h46m13.919388539s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:145.79.227.17:4975): protocol: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection
Feb 26 18:03:08.985159188 (213h46m13.919613328s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection
Feb 26 18:03:08.985363612 (213h46m13.919817753s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12TMMXMjWNxBgccLQMbuExNxgwvzQpNH4cizjaPZDufcjvTqdeh, address:145.79.227.24:4584): protocol: expected piece hash; context deadline exceeded; EOF
Feb 26 18:03:08.985393310 (213h46m13.919847443s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection
Feb 26 18:03:08.985453811 (213h46m13.919907936s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:145.79.227.5:4224): protocol: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection
Feb 26 18:03:08.985487049 (213h46m13.919941172s) [seg_pos={0 0}, upload=188070]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Feb 26 18:03:09.647036800 (213h46m14.581490932s) [upload=188070]: Done uploading segments...
Feb 26 18:03:09.647104855 (213h46m14.581559003s) [upload=188070]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.390074415s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [..X.....X...........X..X..........XX........................] duration:
1.544444661s trace:
https://jaeger.dev.storj.io/trace/3eba5f9dcfb45a38
FAIL (2026-02-26 18:08:18 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.544482683s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
27.263583ms remote:
"tcp:185.244.226.3:443" response headers:
X-Request-Id=8D745kXMbPp X-Xss-Protection=1; mode=block Content-Type=application/xml Server=Storj Date=Thu, 26 Feb 2026 18:08:12 GMT Accept-Ranges=bytes Content-Length=274 Content-Security-Policy=block-all-mixed-content Vary=Origin X-Amz-Request-Id=1897DE33E1E70E1B target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011 trace:
https://jaeger.dev.storj.io/trace/280cb31a67d1537
FAIL (2026-02-26 18:08:12 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE33E1E70E1B</RequestId><HostId></HostId></Error>"
Duration: 27.303422ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
290.064362ms remote:
"tcp:45.38.24.2:443" response headers:
Content-Type=application/xml X-Request-Id=PdgADsWF8Z8 Content-Length=284 Content-Security-Policy=block-all-mixed-content Server=Storj Vary=Origin X-Amz-Request-Id=1897DE32BE8E2AF9 X-Xss-Protection=1; mode=block Date=Thu, 26 Feb 2026 18:08:07 GMT Accept-Ranges=bytes target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/5576317521695ed1
FAIL (2026-02-26 18:08:07 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE32BE8E2AF9</RequestId><HostId></HostId></Error>"
Duration: 290.09798ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
29.569813ms remote:
"tcp:185.244.226.3:443" response headers:
Content-Length=274 Server=Storj Vary=Origin X-Amz-Request-Id=1897DE3149D53613 X-Xss-Protection=1; mode=block Accept-Ranges=bytes Content-Security-Policy=block-all-mixed-content Content-Type=application/xml X-Request-Id=ZRzoQfPSxfH Date=Thu, 26 Feb 2026 18:08:01 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/555db6e1e7508ac4
FAIL (2026-02-26 18:08:01 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE3149D53613</RequestId><HostId></HostId></Error>"
Duration: 29.601072ms
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
2.36592482s trace:
https://jaeger.dev.storj.io/trace/68a06934c0a05f04
FAIL (2026-02-26 18:07:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
eb 26 18:02:40.761285883 (213h45m45.695740025s) [upload=188070]: Starting upload
Feb 26 18:02:40.761395708 (213h45m45.695849854s) [upload=188070]: Done starting upload
Feb 26 18:02:40.761697240 (213h45m45.696151374s) [upload=188070]: Uploading segments...
Feb 26 18:02:40.762148293 (213h45m45.696602435s) [upload=188070]: Got next segment. Inline: false
Feb 26 18:02:40.762222848 (213h45m45.696676987s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 26 18:02:40.762230722 (213h45m45.69668487s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 26 18:02:41.270627701 (213h45m46.20508184s) [seg_pos={0 0}, upload=188070]: Begin upload segment...
Feb 26 18:02:41.270720334 (213h45m46.205174474s) [seg_pos={0 0}, upload=188070]: Done begin upload segment.
Feb 26 18:02:41.270802481 (213h45m46.205256658s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272042027 (213h45m46.206496159s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272283707 (213h45m46.20673785s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272744753 (213h45m46.207198908s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272938814 (213h45m46.207392958s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273277379 (213h45m46.207731498s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273552846 (213h45m46.208006989s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273597171 (213h45m46.208051303s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274079549 (213h45m46.208533671s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274139867 (213h45m46.208594009s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274541012 (213h45m46.208995213s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274823659 (213h45m46.209277801s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.275440186 (213h45m46.209894339s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.276035884 (213h45m46.210490026s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.277403363 (213h45m46.211857506s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.279624637 (213h45m46.214078779s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.889995128 (213h45m46.824449265s) [upload=188070]: Next returned nil segment
Feb 26 18:02:41.890012492 (213h45m46.824466626s) [upload=188070]: Waiting for error group managing segments...
Feb 26 18:02:42.025533148 (213h45m46.959987291s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.026652488 (213h45m46.961106618s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.067448882 (213h45m47.001903023s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.074637015 (213h45m47.009091156s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085003774 (213h45m47.019457898s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085343706 (213h45m47.019797831s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.129629735 (213h45m47.064083879s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.152804779 (213h45m47.087258916s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:55.018263081 (213h45m59.952717215s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:56.602166399 (213h46m1.536620544s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:03.112945430 (213h46m8.047399567s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:08.984934397 (213h46m13.919388539s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:145.79.227.17:4975): protocol: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection
Feb 26 18:03:08.985159188 (213h46m13.919613328s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection
Feb 26 18:03:08.985363612 (213h46m13.919817753s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12TMMXMjWNxBgccLQMbuExNxgwvzQpNH4cizjaPZDufcjvTqdeh, address:145.79.227.24:4584): protocol: expected piece hash; context deadline exceeded; EOF
Feb 26 18:03:08.985393310 (213h46m13.919847443s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection
Feb 26 18:03:08.985453811 (213h46m13.919907936s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:145.79.227.5:4224): protocol: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection
Feb 26 18:03:08.985487049 (213h46m13.919941172s) [seg_pos={0 0}, upload=188070]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Feb 26 18:03:09.647036800 (213h46m14.581490932s) [upload=188070]: Done uploading segments...
Feb 26 18:03:09.647104855 (213h46m14.581559003s) [upload=188070]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.365963418s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
7.99598ms remote:
"tcp:185.244.226.3:443" response headers:
Server=Storj X-Request-Id=3kKHX4suwQR X-Xss-Protection=1; mode=block Date=Thu, 26 Feb 2026 18:07:24 GMT Accept-Ranges=bytes Content-Length=274 Content-Security-Policy=block-all-mixed-content Vary=Origin X-Amz-Request-Id=1897DE289C64D0C1 Content-Type=application/xml target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/1352f00d71f64e50
FAIL (2026-02-26 18:07:24 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE289C64D0C1</RequestId><HostId></HostId></Error>"
Duration: 8.02662ms
edge
gateway
vhost
bgp
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [...X.....X...........X..X..........XX.......................] duration:
2.706363639s trace:
https://jaeger.dev.storj.io/trace/3eba5f9dcfb45a38
FAIL (2026-02-26 18:07:21 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 2.706393017s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
2.542985137s trace:
https://jaeger.dev.storj.io/trace/68a06934c0a05f04
FAIL (2026-02-26 18:07:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
eb 26 18:02:40.761285883 (213h45m45.695740025s) [upload=188070]: Starting upload
Feb 26 18:02:40.761395708 (213h45m45.695849854s) [upload=188070]: Done starting upload
Feb 26 18:02:40.761697240 (213h45m45.696151374s) [upload=188070]: Uploading segments...
Feb 26 18:02:40.762148293 (213h45m45.696602435s) [upload=188070]: Got next segment. Inline: false
Feb 26 18:02:40.762222848 (213h45m45.696676987s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 26 18:02:40.762230722 (213h45m45.69668487s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 26 18:02:41.270627701 (213h45m46.20508184s) [seg_pos={0 0}, upload=188070]: Begin upload segment...
Feb 26 18:02:41.270720334 (213h45m46.205174474s) [seg_pos={0 0}, upload=188070]: Done begin upload segment.
Feb 26 18:02:41.270802481 (213h45m46.205256658s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272042027 (213h45m46.206496159s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272283707 (213h45m46.20673785s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272744753 (213h45m46.207198908s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272938814 (213h45m46.207392958s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273277379 (213h45m46.207731498s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273552846 (213h45m46.208006989s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273597171 (213h45m46.208051303s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274079549 (213h45m46.208533671s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274139867 (213h45m46.208594009s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274541012 (213h45m46.208995213s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274823659 (213h45m46.209277801s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.275440186 (213h45m46.209894339s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.276035884 (213h45m46.210490026s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.277403363 (213h45m46.211857506s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.279624637 (213h45m46.214078779s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.889995128 (213h45m46.824449265s) [upload=188070]: Next returned nil segment
Feb 26 18:02:41.890012492 (213h45m46.824466626s) [upload=188070]: Waiting for error group managing segments...
Feb 26 18:02:42.025533148 (213h45m46.959987291s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.026652488 (213h45m46.961106618s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.067448882 (213h45m47.001903023s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.074637015 (213h45m47.009091156s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085003774 (213h45m47.019457898s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085343706 (213h45m47.019797831s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.129629735 (213h45m47.064083879s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.152804779 (213h45m47.087258916s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:55.018263081 (213h45m59.952717215s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:56.602166399 (213h46m1.536620544s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:03.112945430 (213h46m8.047399567s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:08.984934397 (213h46m13.919388539s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:145.79.227.17:4975): protocol: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection
Feb 26 18:03:08.985159188 (213h46m13.919613328s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection
Feb 26 18:03:08.985363612 (213h46m13.919817753s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12TMMXMjWNxBgccLQMbuExNxgwvzQpNH4cizjaPZDufcjvTqdeh, address:145.79.227.24:4584): protocol: expected piece hash; context deadline exceeded; EOF
Feb 26 18:03:08.985393310 (213h46m13.919847443s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection
Feb 26 18:03:08.985453811 (213h46m13.919907936s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:145.79.227.5:4224): protocol: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection
Feb 26 18:03:08.985487049 (213h46m13.919941172s) [seg_pos={0 0}, upload=188070]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Feb 26 18:03:09.647036800 (213h46m14.581490932s) [upload=188070]: Done uploading segments...
Feb 26 18:03:09.647104855 (213h46m14.581559003s) [upload=188070]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.543034559s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
10.135071ms remote:
"tcp:185.244.226.3:443" response headers:
Content-Type=application/xml X-Amz-Request-Id=1897DE2519CF43EC X-Request-Id=5QzTEtZAte4 X-Xss-Protection=1; mode=block Content-Security-Policy=block-all-mixed-content Server=Storj Vary=Origin Date=Thu, 26 Feb 2026 18:07:09 GMT Accept-Ranges=bytes Content-Length=274 target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/78819a809ee6f7bf
FAIL (2026-02-26 18:07:09 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE2519CF43EC</RequestId><HostId></HostId></Error>"
Duration: 10.151774ms
edge
gateway
vhost
bgp
eu1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
314.668513ms remote:
"tcp:45.38.24.2:443" response headers:
Vary=Origin X-Amz-Request-Id=1897DE2362DEE9AC X-Xss-Protection=1; mode=block Content-Type=application/xml Server=Storj X-Request-Id=AyGke4h5Lnp Date=Thu, 26 Feb 2026 18:07:01 GMT Accept-Ranges=bytes Content-Length=284 Content-Security-Policy=block-all-mixed-content target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/5e49070cd3168335
FAIL (2026-02-26 18:07:02 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE2362DEE9AC</RequestId><HostId></HostId></Error>"
Duration: 314.694137ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
7.768684ms remote:
"tcp:185.244.226.3:443" response headers:
X-Request-Id=ZgHGjo4pzvb Accept-Ranges=bytes Content-Length=274 X-Xss-Protection=1; mode=block Date=Thu, 26 Feb 2026 18:06:58 GMT Content-Security-Policy=block-all-mixed-content Content-Type=application/xml Server=Storj Vary=Origin X-Amz-Request-Id=1897DE227E05A58A target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011 trace:
https://jaeger.dev.storj.io/trace/7bf4608cc88ffab7
FAIL (2026-02-26 18:06:58 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE227E05A58A</RequestId><HostId></HostId></Error>"
Duration: 7.792587ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
12.294786ms remote:
"tcp:185.244.226.3:443" response headers:
Content-Length=274 Content-Security-Policy=block-all-mixed-content Server=Storj X-Amz-Request-Id=1897DE21943D46D1 Content-Type=application/xml Vary=Origin X-Request-Id=Ue2r7LVdhZY X-Xss-Protection=1; mode=block Date=Thu, 26 Feb 2026 18:06:54 GMT Accept-Ranges=bytes target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/5cff08478a7feb1a
FAIL (2026-02-26 18:06:54 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE21943D46D1</RequestId><HostId></HostId></Error>"
Duration: 12.390174ms
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
2.481126252s trace:
https://jaeger.dev.storj.io/trace/68a06934c0a05f04
FAIL (2026-02-26 18:06:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
eb 26 18:02:40.761285883 (213h45m45.695740025s) [upload=188070]: Starting upload
Feb 26 18:02:40.761395708 (213h45m45.695849854s) [upload=188070]: Done starting upload
Feb 26 18:02:40.761697240 (213h45m45.696151374s) [upload=188070]: Uploading segments...
Feb 26 18:02:40.762148293 (213h45m45.696602435s) [upload=188070]: Got next segment. Inline: false
Feb 26 18:02:40.762222848 (213h45m45.696676987s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 26 18:02:40.762230722 (213h45m45.69668487s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 26 18:02:41.270627701 (213h45m46.20508184s) [seg_pos={0 0}, upload=188070]: Begin upload segment...
Feb 26 18:02:41.270720334 (213h45m46.205174474s) [seg_pos={0 0}, upload=188070]: Done begin upload segment.
Feb 26 18:02:41.270802481 (213h45m46.205256658s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272042027 (213h45m46.206496159s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272283707 (213h45m46.20673785s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272744753 (213h45m46.207198908s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272938814 (213h45m46.207392958s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273277379 (213h45m46.207731498s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273552846 (213h45m46.208006989s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273597171 (213h45m46.208051303s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274079549 (213h45m46.208533671s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274139867 (213h45m46.208594009s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274541012 (213h45m46.208995213s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274823659 (213h45m46.209277801s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.275440186 (213h45m46.209894339s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.276035884 (213h45m46.210490026s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.277403363 (213h45m46.211857506s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.279624637 (213h45m46.214078779s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.889995128 (213h45m46.824449265s) [upload=188070]: Next returned nil segment
Feb 26 18:02:41.890012492 (213h45m46.824466626s) [upload=188070]: Waiting for error group managing segments...
Feb 26 18:02:42.025533148 (213h45m46.959987291s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.026652488 (213h45m46.961106618s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.067448882 (213h45m47.001903023s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.074637015 (213h45m47.009091156s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085003774 (213h45m47.019457898s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085343706 (213h45m47.019797831s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.129629735 (213h45m47.064083879s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.152804779 (213h45m47.087258916s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:55.018263081 (213h45m59.952717215s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:56.602166399 (213h46m1.536620544s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:03.112945430 (213h46m8.047399567s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:08.984934397 (213h46m13.919388539s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:145.79.227.17:4975): protocol: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection
Feb 26 18:03:08.985159188 (213h46m13.919613328s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection
Feb 26 18:03:08.985363612 (213h46m13.919817753s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12TMMXMjWNxBgccLQMbuExNxgwvzQpNH4cizjaPZDufcjvTqdeh, address:145.79.227.24:4584): protocol: expected piece hash; context deadline exceeded; EOF
Feb 26 18:03:08.985393310 (213h46m13.919847443s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection
Feb 26 18:03:08.985453811 (213h46m13.919907936s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:145.79.227.5:4224): protocol: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection
Feb 26 18:03:08.985487049 (213h46m13.919941172s) [seg_pos={0 0}, upload=188070]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Feb 26 18:03:09.647036800 (213h46m14.581490932s) [upload=188070]: Done uploading segments...
Feb 26 18:03:09.647104855 (213h46m14.581559003s) [upload=188070]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.481238455s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [....X.....X...........X..X..........XX......................] duration:
1.932406527s trace:
https://jaeger.dev.storj.io/trace/3eba5f9dcfb45a38
FAIL (2026-02-26 18:06:29 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.932438013s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
9.049926ms remote:
"tcp:185.244.226.3:443" response headers:
Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=d3FysZ3DyJq X-Xss-Protection=1; mode=block Accept-Ranges=bytes Content-Length=274 Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1897DE185773B238 Date=Thu, 26 Feb 2026 18:06:14 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/3e6830fe17c5a043
FAIL (2026-02-26 18:06:14 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE185773B238</RequestId><HostId></HostId></Error>"
Duration: 9.091156ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
4.34943194s trace:
https://jaeger.dev.storj.io/trace/68a06934c0a05f04
FAIL (2026-02-26 18:06:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
eb 26 18:02:40.761285883 (213h45m45.695740025s) [upload=188070]: Starting upload
Feb 26 18:02:40.761395708 (213h45m45.695849854s) [upload=188070]: Done starting upload
Feb 26 18:02:40.761697240 (213h45m45.696151374s) [upload=188070]: Uploading segments...
Feb 26 18:02:40.762148293 (213h45m45.696602435s) [upload=188070]: Got next segment. Inline: false
Feb 26 18:02:40.762222848 (213h45m45.696676987s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 26 18:02:40.762230722 (213h45m45.69668487s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 26 18:02:41.270627701 (213h45m46.20508184s) [seg_pos={0 0}, upload=188070]: Begin upload segment...
Feb 26 18:02:41.270720334 (213h45m46.205174474s) [seg_pos={0 0}, upload=188070]: Done begin upload segment.
Feb 26 18:02:41.270802481 (213h45m46.205256658s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272042027 (213h45m46.206496159s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272283707 (213h45m46.20673785s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272744753 (213h45m46.207198908s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272938814 (213h45m46.207392958s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273277379 (213h45m46.207731498s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273552846 (213h45m46.208006989s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273597171 (213h45m46.208051303s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274079549 (213h45m46.208533671s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274139867 (213h45m46.208594009s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274541012 (213h45m46.208995213s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274823659 (213h45m46.209277801s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.275440186 (213h45m46.209894339s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.276035884 (213h45m46.210490026s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.277403363 (213h45m46.211857506s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.279624637 (213h45m46.214078779s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.889995128 (213h45m46.824449265s) [upload=188070]: Next returned nil segment
Feb 26 18:02:41.890012492 (213h45m46.824466626s) [upload=188070]: Waiting for error group managing segments...
Feb 26 18:02:42.025533148 (213h45m46.959987291s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.026652488 (213h45m46.961106618s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.067448882 (213h45m47.001903023s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.074637015 (213h45m47.009091156s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085003774 (213h45m47.019457898s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085343706 (213h45m47.019797831s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.129629735 (213h45m47.064083879s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.152804779 (213h45m47.087258916s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:55.018263081 (213h45m59.952717215s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:56.602166399 (213h46m1.536620544s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:03.112945430 (213h46m8.047399567s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:08.984934397 (213h46m13.919388539s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:145.79.227.17:4975): protocol: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection
Feb 26 18:03:08.985159188 (213h46m13.919613328s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection
Feb 26 18:03:08.985363612 (213h46m13.919817753s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12TMMXMjWNxBgccLQMbuExNxgwvzQpNH4cizjaPZDufcjvTqdeh, address:145.79.227.24:4584): protocol: expected piece hash; context deadline exceeded; EOF
Feb 26 18:03:08.985393310 (213h46m13.919847443s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection
Feb 26 18:03:08.985453811 (213h46m13.919907936s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:145.79.227.5:4224): protocol: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection
Feb 26 18:03:08.985487049 (213h46m13.919941172s) [seg_pos={0 0}, upload=188070]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Feb 26 18:03:09.647036800 (213h46m14.581490932s) [upload=188070]: Done uploading segments...
Feb 26 18:03:09.647104855 (213h46m14.581559003s) [upload=188070]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 4.349465079s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
292.315394ms remote:
"tcp:45.38.24.2:443" response headers:
Content-Length=284 Content-Type=application/xml Server=Storj Vary=Origin X-Amz-Request-Id=1897DE15477075FA X-Request-Id=b8vucDfEBP1 Date=Thu, 26 Feb 2026 18:06:01 GMT Content-Security-Policy=block-all-mixed-content X-Xss-Protection=1; mode=block Accept-Ranges=bytes target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/4167e7e9adc6a63b
FAIL (2026-02-26 18:06:01 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE15477075FA</RequestId><HostId></HostId></Error>"
Duration: 292.374889ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
20.475424ms remote:
"tcp:185.244.226.3:443" response headers:
Content-Security-Policy=block-all-mixed-content Server=Storj Vary=Origin X-Amz-Request-Id=1897DE150A5E67B0 X-Request-Id=3RrNNYHRoCS Date=Thu, 26 Feb 2026 18:06:00 GMT Content-Type=application/xml X-Xss-Protection=1; mode=block Accept-Ranges=bytes Content-Length=274 target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011 trace:
https://jaeger.dev.storj.io/trace/516f18dbb701260b
FAIL (2026-02-26 18:06:00 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE150A5E67B0</RequestId><HostId></HostId></Error>"
Duration: 20.502031ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
9.660822ms remote:
"tcp:185.244.226.3:443" response headers:
Content-Length=274 Content-Security-Policy=block-all-mixed-content Content-Type=application/xml X-Request-Id=4YABfwo1ddA Accept-Ranges=bytes Server=Storj Vary=Origin X-Amz-Request-Id=1897DE11D9BD5D1D X-Xss-Protection=1; mode=block Date=Thu, 26 Feb 2026 18:05:46 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/587efde7e1c5cb98
FAIL (2026-02-26 18:05:46 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE11D9BD5D1D</RequestId><HostId></HostId></Error>"
Duration: 9.808585ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [.....X.....X...........X..X..........XX.....................] duration:
1.679296266s trace:
https://jaeger.dev.storj.io/trace/3eba5f9dcfb45a38
FAIL (2026-02-26 18:05:38 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.679339684s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
14.464707ms remote:
"tcp:185.244.226.3:443" response headers:
X-Amz-Request-Id=1897DE0F75566587 Date=Thu, 26 Feb 2026 18:05:36 GMT Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=GzehKhnnSq5 X-Xss-Protection=1; mode=block Accept-Ranges=bytes Content-Length=274 Content-Type=application/xml Vary=Origin target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/7232a3da7c947792
FAIL (2026-02-26 18:05:36 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE0F75566587</RequestId><HostId></HostId></Error>"
Duration: 14.545019ms
edge
gateway
vhost
bgp
eu1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
8.412476ms remote:
"tcp:185.244.226.3:443" response headers:
X-Request-Id=J9XhM1H1rah X-Xss-Protection=1; mode=block Date=Thu, 26 Feb 2026 18:05:08 GMT Accept-Ranges=bytes Content-Length=274 Content-Security-Policy=block-all-mixed-content Content-Type=application/xml Server=Storj Vary=Origin X-Amz-Request-Id=1897DE08F8320F2B target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011 trace:
https://jaeger.dev.storj.io/trace/569e3340cad39f5e
FAIL (2026-02-26 18:05:08 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE08F8320F2B</RequestId><HostId></HostId></Error>"
Duration: 8.456776ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
2.390891995s trace:
https://jaeger.dev.storj.io/trace/68a06934c0a05f04
FAIL (2026-02-26 18:05:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
eb 26 18:02:40.761285883 (213h45m45.695740025s) [upload=188070]: Starting upload
Feb 26 18:02:40.761395708 (213h45m45.695849854s) [upload=188070]: Done starting upload
Feb 26 18:02:40.761697240 (213h45m45.696151374s) [upload=188070]: Uploading segments...
Feb 26 18:02:40.762148293 (213h45m45.696602435s) [upload=188070]: Got next segment. Inline: false
Feb 26 18:02:40.762222848 (213h45m45.696676987s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 26 18:02:40.762230722 (213h45m45.69668487s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 26 18:02:41.270627701 (213h45m46.20508184s) [seg_pos={0 0}, upload=188070]: Begin upload segment...
Feb 26 18:02:41.270720334 (213h45m46.205174474s) [seg_pos={0 0}, upload=188070]: Done begin upload segment.
Feb 26 18:02:41.270802481 (213h45m46.205256658s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272042027 (213h45m46.206496159s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272283707 (213h45m46.20673785s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272744753 (213h45m46.207198908s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272938814 (213h45m46.207392958s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273277379 (213h45m46.207731498s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273552846 (213h45m46.208006989s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273597171 (213h45m46.208051303s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274079549 (213h45m46.208533671s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274139867 (213h45m46.208594009s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274541012 (213h45m46.208995213s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274823659 (213h45m46.209277801s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.275440186 (213h45m46.209894339s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.276035884 (213h45m46.210490026s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.277403363 (213h45m46.211857506s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.279624637 (213h45m46.214078779s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.889995128 (213h45m46.824449265s) [upload=188070]: Next returned nil segment
Feb 26 18:02:41.890012492 (213h45m46.824466626s) [upload=188070]: Waiting for error group managing segments...
Feb 26 18:02:42.025533148 (213h45m46.959987291s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.026652488 (213h45m46.961106618s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.067448882 (213h45m47.001903023s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.074637015 (213h45m47.009091156s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085003774 (213h45m47.019457898s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085343706 (213h45m47.019797831s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.129629735 (213h45m47.064083879s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.152804779 (213h45m47.087258916s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:55.018263081 (213h45m59.952717215s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:56.602166399 (213h46m1.536620544s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:03.112945430 (213h46m8.047399567s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:08.984934397 (213h46m13.919388539s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:145.79.227.17:4975): protocol: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection
Feb 26 18:03:08.985159188 (213h46m13.919613328s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection
Feb 26 18:03:08.985363612 (213h46m13.919817753s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12TMMXMjWNxBgccLQMbuExNxgwvzQpNH4cizjaPZDufcjvTqdeh, address:145.79.227.24:4584): protocol: expected piece hash; context deadline exceeded; EOF
Feb 26 18:03:08.985393310 (213h46m13.919847443s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection
Feb 26 18:03:08.985453811 (213h46m13.919907936s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:145.79.227.5:4224): protocol: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection
Feb 26 18:03:08.985487049 (213h46m13.919941172s) [seg_pos={0 0}, upload=188070]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Feb 26 18:03:09.647036800 (213h46m14.581490932s) [upload=188070]: Done uploading segments...
Feb 26 18:03:09.647104855 (213h46m14.581559003s) [upload=188070]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.39093097s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
304.486844ms remote:
"tcp:45.38.24.2:443" response headers:
Content-Length=284 Content-Security-Policy=block-all-mixed-content Server=Storj Vary=Origin Accept-Ranges=bytes Content-Type=application/xml X-Amz-Request-Id=1897DE06F93DEEBF X-Request-Id=c7AUQv7ymNz X-Xss-Protection=1; mode=block Date=Thu, 26 Feb 2026 18:04:59 GMT target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/3075392e274b5316
FAIL (2026-02-26 18:04:59 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE06F93DEEBF</RequestId><HostId></HostId></Error>"
Duration: 304.516565ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
8.604484ms remote:
"tcp:185.244.226.3:443" response headers:
Content-Type=application/xml Server=Storj Vary=Origin X-Amz-Request-Id=1897DE05BDCE0EA4 Date=Thu, 26 Feb 2026 18:04:54 GMT Accept-Ranges=bytes Content-Length=274 Content-Security-Policy=block-all-mixed-content X-Request-Id=6uzRzvnJZHK X-Xss-Protection=1; mode=block target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/4e9ee3fcb1078aaa
FAIL (2026-02-26 18:04:54 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE05BDCE0EA4</RequestId><HostId></HostId></Error>"
Duration: 8.631205ms
edge
gateway
vhost
bgp
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [X.....X.....X...........X..X..........XX....................] duration:
1.685269333s trace:
https://jaeger.dev.storj.io/trace/3eba5f9dcfb45a38
FAIL (2026-02-26 18:04:53 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.685299321s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
2.418183667s trace:
https://jaeger.dev.storj.io/trace/68a06934c0a05f04
FAIL (2026-02-26 18:04:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
eb 26 18:02:40.761285883 (213h45m45.695740025s) [upload=188070]: Starting upload
Feb 26 18:02:40.761395708 (213h45m45.695849854s) [upload=188070]: Done starting upload
Feb 26 18:02:40.761697240 (213h45m45.696151374s) [upload=188070]: Uploading segments...
Feb 26 18:02:40.762148293 (213h45m45.696602435s) [upload=188070]: Got next segment. Inline: false
Feb 26 18:02:40.762222848 (213h45m45.696676987s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 26 18:02:40.762230722 (213h45m45.69668487s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 26 18:02:41.270627701 (213h45m46.20508184s) [seg_pos={0 0}, upload=188070]: Begin upload segment...
Feb 26 18:02:41.270720334 (213h45m46.205174474s) [seg_pos={0 0}, upload=188070]: Done begin upload segment.
Feb 26 18:02:41.270802481 (213h45m46.205256658s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272042027 (213h45m46.206496159s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272283707 (213h45m46.20673785s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272744753 (213h45m46.207198908s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272938814 (213h45m46.207392958s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273277379 (213h45m46.207731498s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273552846 (213h45m46.208006989s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273597171 (213h45m46.208051303s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274079549 (213h45m46.208533671s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274139867 (213h45m46.208594009s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274541012 (213h45m46.208995213s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274823659 (213h45m46.209277801s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.275440186 (213h45m46.209894339s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.276035884 (213h45m46.210490026s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.277403363 (213h45m46.211857506s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.279624637 (213h45m46.214078779s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.889995128 (213h45m46.824449265s) [upload=188070]: Next returned nil segment
Feb 26 18:02:41.890012492 (213h45m46.824466626s) [upload=188070]: Waiting for error group managing segments...
Feb 26 18:02:42.025533148 (213h45m46.959987291s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.026652488 (213h45m46.961106618s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.067448882 (213h45m47.001903023s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.074637015 (213h45m47.009091156s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085003774 (213h45m47.019457898s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085343706 (213h45m47.019797831s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.129629735 (213h45m47.064083879s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.152804779 (213h45m47.087258916s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:55.018263081 (213h45m59.952717215s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:56.602166399 (213h46m1.536620544s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:03.112945430 (213h46m8.047399567s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:08.984934397 (213h46m13.919388539s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:145.79.227.17:4975): protocol: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection
Feb 26 18:03:08.985159188 (213h46m13.919613328s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection
Feb 26 18:03:08.985363612 (213h46m13.919817753s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12TMMXMjWNxBgccLQMbuExNxgwvzQpNH4cizjaPZDufcjvTqdeh, address:145.79.227.24:4584): protocol: expected piece hash; context deadline exceeded; EOF
Feb 26 18:03:08.985393310 (213h46m13.919847443s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection
Feb 26 18:03:08.985453811 (213h46m13.919907936s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:145.79.227.5:4224): protocol: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection
Feb 26 18:03:08.985487049 (213h46m13.919941172s) [seg_pos={0 0}, upload=188070]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Feb 26 18:03:09.647036800 (213h46m14.581490932s) [upload=188070]: Done uploading segments...
Feb 26 18:03:09.647104855 (213h46m14.581559003s) [upload=188070]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.418224932s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
288.115967ms remote:
"tcp:45.38.24.2:443" response headers:
Content-Length=284 Content-Security-Policy=block-all-mixed-content Server=Storj Vary=Origin X-Amz-Request-Id=1897DE01E2E96BA0 Date=Thu, 26 Feb 2026 18:04:38 GMT Accept-Ranges=bytes Content-Type=application/xml X-Request-Id=7nQ9NSiJX2g X-Xss-Protection=1; mode=block target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/8a10f9644de3d42
FAIL (2026-02-26 18:04:38 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE01E2E96BA0</RequestId><HostId></HostId></Error>"
Duration: 288.23769ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
7.58251ms remote:
"tcp:185.244.226.3:443" response headers:
Content-Length=274 Content-Security-Policy=block-all-mixed-content Server=Storj Vary=Origin X-Amz-Request-Id=1897DE00ED0344C6 X-Request-Id=6VyeQsSsVo8 X-Xss-Protection=1; mode=block Accept-Ranges=bytes Content-Type=application/xml Date=Thu, 26 Feb 2026 18:04:33 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/e84166bc500d953
FAIL (2026-02-26 18:04:33 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE00ED0344C6</RequestId><HostId></HostId></Error>"
Duration: 7.639649ms
edge
gateway
vhost
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
12.27198ms remote:
"tcp:185.244.226.3:443" response headers:
X-Xss-Protection=1; mode=block Date=Thu, 26 Feb 2026 18:04:30 GMT Accept-Ranges=bytes Content-Security-Policy=block-all-mixed-content X-Request-Id=NDL8f3p9bc Content-Length=274 Content-Type=application/xml Server=Storj Vary=Origin X-Amz-Request-Id=1897DE0035660AD4 target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011 trace:
https://jaeger.dev.storj.io/trace/58ad44eada7b126
FAIL (2026-02-26 18:04:30 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DE0035660AD4</RequestId><HostId></HostId></Error>"
Duration: 12.302686ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
711.253975ms remote:
"tcp:45.38.24.2:443" response headers:
Content-Type=application/xml Server=Storj Vary=Origin X-Amz-Request-Id=1897DDFA02E85985 X-Xss-Protection=1; mode=block X-Request-Id=fmwuhBnoZpr Date=Thu, 26 Feb 2026 18:04:04 GMT Accept-Ranges=bytes Content-Length=284 Content-Security-Policy=block-all-mixed-content target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/1ad7a401cbea306c
FAIL (2026-02-26 18:04:04 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DDFA02E85985</RequestId><HostId></HostId></Error>"
Duration: 711.291996ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
8.108005ms remote:
"tcp:185.244.226.3:443" response headers:
Server=Storj Vary=Origin X-Amz-Request-Id=1897DDF8930563AD X-Request-Id=9LZAUQaXrDy X-Xss-Protection=1; mode=block Content-Security-Policy=block-all-mixed-content Date=Thu, 26 Feb 2026 18:03:58 GMT Accept-Ranges=bytes Content-Length=274 Content-Type=application/xml target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/4665539b5b5258b4
FAIL (2026-02-26 18:03:58 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DDF8930563AD</RequestId><HostId></HostId></Error>"
Duration: 8.129239ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.354382591s trace:
https://jaeger.dev.storj.io/trace/68a06934c0a05f04
FAIL (2026-02-26 18:03:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
eb 26 18:02:40.761285883 (213h45m45.695740025s) [upload=188070]: Starting upload
Feb 26 18:02:40.761395708 (213h45m45.695849854s) [upload=188070]: Done starting upload
Feb 26 18:02:40.761697240 (213h45m45.696151374s) [upload=188070]: Uploading segments...
Feb 26 18:02:40.762148293 (213h45m45.696602435s) [upload=188070]: Got next segment. Inline: false
Feb 26 18:02:40.762222848 (213h45m45.696676987s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 26 18:02:40.762230722 (213h45m45.69668487s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 26 18:02:41.270627701 (213h45m46.20508184s) [seg_pos={0 0}, upload=188070]: Begin upload segment...
Feb 26 18:02:41.270720334 (213h45m46.205174474s) [seg_pos={0 0}, upload=188070]: Done begin upload segment.
Feb 26 18:02:41.270802481 (213h45m46.205256658s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272042027 (213h45m46.206496159s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272283707 (213h45m46.20673785s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272744753 (213h45m46.207198908s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272938814 (213h45m46.207392958s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273277379 (213h45m46.207731498s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273552846 (213h45m46.208006989s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273597171 (213h45m46.208051303s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274079549 (213h45m46.208533671s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274139867 (213h45m46.208594009s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274541012 (213h45m46.208995213s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274823659 (213h45m46.209277801s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.275440186 (213h45m46.209894339s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.276035884 (213h45m46.210490026s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.277403363 (213h45m46.211857506s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.279624637 (213h45m46.214078779s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.889995128 (213h45m46.824449265s) [upload=188070]: Next returned nil segment
Feb 26 18:02:41.890012492 (213h45m46.824466626s) [upload=188070]: Waiting for error group managing segments...
Feb 26 18:02:42.025533148 (213h45m46.959987291s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.026652488 (213h45m46.961106618s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.067448882 (213h45m47.001903023s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.074637015 (213h45m47.009091156s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085003774 (213h45m47.019457898s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085343706 (213h45m47.019797831s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.129629735 (213h45m47.064083879s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.152804779 (213h45m47.087258916s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:55.018263081 (213h45m59.952717215s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:56.602166399 (213h46m1.536620544s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:03.112945430 (213h46m8.047399567s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:08.984934397 (213h46m13.919388539s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:145.79.227.17:4975): protocol: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection
Feb 26 18:03:08.985159188 (213h46m13.919613328s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection
Feb 26 18:03:08.985363612 (213h46m13.919817753s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12TMMXMjWNxBgccLQMbuExNxgwvzQpNH4cizjaPZDufcjvTqdeh, address:145.79.227.24:4584): protocol: expected piece hash; context deadline exceeded; EOF
Feb 26 18:03:08.985393310 (213h46m13.919847443s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection
Feb 26 18:03:08.985453811 (213h46m13.919907936s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:145.79.227.5:4224): protocol: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection
Feb 26 18:03:08.985487049 (213h46m13.919941172s) [seg_pos={0 0}, upload=188070]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Feb 26 18:03:09.647036800 (213h46m14.581490932s) [upload=188070]: Done uploading segments...
Feb 26 18:03:09.647104855 (213h46m14.581559003s) [upload=188070]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.354413922s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.X.....X.....X...........X..X..........XX...................] duration:
1.617004297s trace:
https://jaeger.dev.storj.io/trace/3eba5f9dcfb45a38
FAIL (2026-02-26 18:03:50 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.617047276s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
289.385873ms remote:
"tcp:45.38.24.2:443" response headers:
Date=Thu, 26 Feb 2026 18:03:45 GMT Accept-Ranges=bytes X-Amz-Request-Id=1897DDF5BD03B9C6 Content-Length=284 Content-Security-Policy=block-all-mixed-content Content-Type=application/xml Server=Storj Vary=Origin X-Request-Id=KWoyyhWeChy X-Xss-Protection=1; mode=block target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/226134612715db76
FAIL (2026-02-26 18:03:45 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DDF5BD03B9C6</RequestId><HostId></HostId></Error>"
Duration: 289.410772ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
8.360504ms remote:
"tcp:185.244.226.3:443" response headers:
X-Amz-Request-Id=1897DDF3A84397A8 Accept-Ranges=bytes Server=Storj X-Request-Id=Wfgi6UeY8dt X-Xss-Protection=1; mode=block Date=Thu, 26 Feb 2026 18:03:36 GMT Content-Length=274 Content-Security-Policy=block-all-mixed-content Content-Type=application/xml Vary=Origin target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011 trace:
https://jaeger.dev.storj.io/trace/70e095c8b3149977
FAIL (2026-02-26 18:03:36 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DDF3A84397A8</RequestId><HostId></HostId></Error>"
Duration: 8.385956ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
10.458689ms remote:
"tcp:185.244.226.3:443" response headers:
X-Amz-Request-Id=1897DDF29E0F6181 Date=Thu, 26 Feb 2026 18:03:32 GMT Accept-Ranges=bytes Content-Length=274 Server=Storj Vary=Origin X-Request-Id=Qiv5L2ue5Yb X-Xss-Protection=1; mode=block Content-Security-Policy=block-all-mixed-content Content-Type=application/xml target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/7dea06065f316138
FAIL (2026-02-26 18:03:32 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DDF29E0F6181</RequestId><HostId></HostId></Error>"
Duration: 10.517798ms
edge
gateway
vhost
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
12.814604ms remote:
"tcp:185.244.226.3:443" response headers:
Accept-Ranges=bytes Content-Length=274 Content-Security-Policy=block-all-mixed-content X-Amz-Request-Id=1897DDF230E298AA Date=Thu, 26 Feb 2026 18:03:30 GMT Content-Type=application/xml Server=Storj Vary=Origin X-Request-Id=D6ybeDQHFiv X-Xss-Protection=1; mode=block target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/45ed9e7f63cea120
FAIL (2026-02-26 18:03:30 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DDF230E298AA</RequestId><HostId></HostId></Error>"
Duration: 12.925616ms
edge
gateway
vhost
bgp
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [..X.....X.....X...........X..X..........XX..................] duration:
2.345701007s trace:
https://jaeger.dev.storj.io/trace/3eba5f9dcfb45a38
FAIL (2026-02-26 18:03:25 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 2.345730324s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
2.750980902s trace:
https://jaeger.dev.storj.io/trace/68a06934c0a05f04
FAIL (2026-02-26 18:03:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
eb 26 18:02:40.761285883 (213h45m45.695740025s) [upload=188070]: Starting upload
Feb 26 18:02:40.761395708 (213h45m45.695849854s) [upload=188070]: Done starting upload
Feb 26 18:02:40.761697240 (213h45m45.696151374s) [upload=188070]: Uploading segments...
Feb 26 18:02:40.762148293 (213h45m45.696602435s) [upload=188070]: Got next segment. Inline: false
Feb 26 18:02:40.762222848 (213h45m45.696676987s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 26 18:02:40.762230722 (213h45m45.69668487s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 26 18:02:41.270627701 (213h45m46.20508184s) [seg_pos={0 0}, upload=188070]: Begin upload segment...
Feb 26 18:02:41.270720334 (213h45m46.205174474s) [seg_pos={0 0}, upload=188070]: Done begin upload segment.
Feb 26 18:02:41.270802481 (213h45m46.205256658s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272042027 (213h45m46.206496159s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272283707 (213h45m46.20673785s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272744753 (213h45m46.207198908s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272938814 (213h45m46.207392958s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273277379 (213h45m46.207731498s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273552846 (213h45m46.208006989s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273597171 (213h45m46.208051303s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274079549 (213h45m46.208533671s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274139867 (213h45m46.208594009s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274541012 (213h45m46.208995213s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274823659 (213h45m46.209277801s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.275440186 (213h45m46.209894339s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.276035884 (213h45m46.210490026s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.277403363 (213h45m46.211857506s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.279624637 (213h45m46.214078779s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.889995128 (213h45m46.824449265s) [upload=188070]: Next returned nil segment
Feb 26 18:02:41.890012492 (213h45m46.824466626s) [upload=188070]: Waiting for error group managing segments...
Feb 26 18:02:42.025533148 (213h45m46.959987291s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.026652488 (213h45m46.961106618s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.067448882 (213h45m47.001903023s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.074637015 (213h45m47.009091156s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085003774 (213h45m47.019457898s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085343706 (213h45m47.019797831s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.129629735 (213h45m47.064083879s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.152804779 (213h45m47.087258916s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:55.018263081 (213h45m59.952717215s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:56.602166399 (213h46m1.536620544s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:03.112945430 (213h46m8.047399567s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:08.984934397 (213h46m13.919388539s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:145.79.227.17:4975): protocol: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection
Feb 26 18:03:08.985159188 (213h46m13.919613328s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection
Feb 26 18:03:08.985363612 (213h46m13.919817753s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12TMMXMjWNxBgccLQMbuExNxgwvzQpNH4cizjaPZDufcjvTqdeh, address:145.79.227.24:4584): protocol: expected piece hash; context deadline exceeded; EOF
Feb 26 18:03:08.985393310 (213h46m13.919847443s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection
Feb 26 18:03:08.985453811 (213h46m13.919907936s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:145.79.227.5:4224): protocol: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection
Feb 26 18:03:08.985487049 (213h46m13.919941172s) [seg_pos={0 0}, upload=188070]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Feb 26 18:03:09.647036800 (213h46m14.581490932s) [upload=188070]: Done uploading segments...
Feb 26 18:03:09.647104855 (213h46m14.581559003s) [upload=188070]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.751007158s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.664129339s trace:
https://jaeger.dev.storj.io/trace/68a06934c0a05f04
FAIL (2026-02-26 18:03:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
eb 26 18:02:40.761285883 (213h45m45.695740025s) [upload=188070]: Starting upload
Feb 26 18:02:40.761395708 (213h45m45.695849854s) [upload=188070]: Done starting upload
Feb 26 18:02:40.761697240 (213h45m45.696151374s) [upload=188070]: Uploading segments...
Feb 26 18:02:40.762148293 (213h45m45.696602435s) [upload=188070]: Got next segment. Inline: false
Feb 26 18:02:40.762222848 (213h45m45.696676987s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 26 18:02:40.762230722 (213h45m45.69668487s) [upload=188070]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 26 18:02:41.270627701 (213h45m46.20508184s) [seg_pos={0 0}, upload=188070]: Begin upload segment...
Feb 26 18:02:41.270720334 (213h45m46.205174474s) [seg_pos={0 0}, upload=188070]: Done begin upload segment.
Feb 26 18:02:41.270802481 (213h45m46.205256658s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272042027 (213h45m46.206496159s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272283707 (213h45m46.20673785s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272744753 (213h45m46.207198908s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.272938814 (213h45m46.207392958s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273277379 (213h45m46.207731498s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273552846 (213h45m46.208006989s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.273597171 (213h45m46.208051303s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274079549 (213h45m46.208533671s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274139867 (213h45m46.208594009s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274541012 (213h45m46.208995213s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.274823659 (213h45m46.209277801s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.275440186 (213h45m46.209894339s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.276035884 (213h45m46.210490026s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.277403363 (213h45m46.211857506s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.279624637 (213h45m46.214078779s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Uploading piece...
Feb 26 18:02:41.889995128 (213h45m46.824449265s) [upload=188070]: Next returned nil segment
Feb 26 18:02:41.890012492 (213h45m46.824466626s) [upload=188070]: Waiting for error group managing segments...
Feb 26 18:02:42.025533148 (213h45m46.959987291s) [noise=true , address=45.91.116.14:4196 , piece_id=HCNADG4RROT5POVVKVRRLPG47GSJNTKKPN4VKZFZLRAQIA2AGQGA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.026652488 (213h45m46.961106618s) [noise=true , address=45.91.116.12:4260 , piece_id=2JAUJW744WIAF62XHFXYLVETLQKTPB7JPZWL2OYFOJJBHIQ4YEAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.067448882 (213h45m47.001903023s) [noise=true , address=145.79.227.27:4692 , piece_id=J3DCU4JTRRKRHXSITWSREXSLAC457LHYKH3WAKWY3676LQNRLBIQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.074637015 (213h45m47.009091156s) [noise=true , address=145.79.227.49:4340 , piece_id=4DY3PUAA3YK64RAGQJG7AQL7A4326N4JS3EPF2IWLXS374JFLD2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085003774 (213h45m47.019457898s) [noise=true , address=145.79.227.34:4909 , piece_id=FBBP5DIR225WLVV3ZOOINTY6V6BW4IUSU4XAWFMBGCPX2VJ5QNFA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.085343706 (213h45m47.019797831s) [noise=true , address=145.79.227.18:4007 , piece_id=B74HKCRWM5O4UBJHC4SUCTCCB234ZJXXKSRB7GDLLS2BA2KEKYAA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.129629735 (213h45m47.064083879s) [noise=true , address=145.79.227.7:4298 , piece_id=3MZOHCVKY6RJQFI3YBRQI4N7QD5HQOEQGMETMBSQVHZAOZPPCU2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:42.152804779 (213h45m47.087258916s) [noise=true , address=77.90.31.19:4842 , piece_id=LTJK4F4OQJIW3JP3QGX4RXWOW2RXQCGAUJV3PJSRQVWOYTSDQHVQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:55.018263081 (213h45m59.952717215s) [noise=true , address=145.79.227.28:4727 , piece_id=ANJU4GLFZ2CMCNUDPHKTBIPAQRU67QKHIRBBXTYEN2YKXNJFN5NA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:02:56.602166399 (213h46m1.536620544s) [noise=true , address=145.79.227.39:4013 , piece_id=HF6BOIHHMPC6SVI2KVBLOAK5PE5AUDASGPFV6XGX2EVBQF262R2Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:03.112945430 (213h46m8.047399567s) [noise=true , address=145.79.227.16:4911 , piece_id=R7XS2TLYUBGTWKTV7WYJASYGPIS4F2IIOIO6AM2O345OK2PVNR2A, seg_pos={0 0}, upload=188070]: Done uploading piece. err: <nil>
Feb 26 18:03:08.984934397 (213h46m13.919388539s) [noise=true , address=145.79.227.17:4975 , piece_id=WHAFQWHPO3P6CJXK2UTQIVUI5XW5AY4MVGPE6N6IEU2RNHMF7DHA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:145.79.227.17:4975): protocol: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60550->145.79.227.17:4975: use of closed network connection
Feb 26 18:03:08.985159188 (213h46m13.919613328s) [noise=true , address=145.79.227.38:4982 , piece_id=CDJZME7DQXICXLJNZUGUG6TCT7MD7MOMIZLNPNGNW5RJLQ3E3Q4Q, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34792->145.79.227.38:4982: use of closed network connection
Feb 26 18:03:08.985363612 (213h46m13.919817753s) [noise=true , address=145.79.227.24:4584 , piece_id=5WWPMU7BO3WLX5A6SAOYMFTO2YDVAAB6TLBVUBOUDAJMRT33OLYQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12TMMXMjWNxBgccLQMbuExNxgwvzQpNH4cizjaPZDufcjvTqdeh, address:145.79.227.24:4584): protocol: expected piece hash; context deadline exceeded; EOF
Feb 26 18:03:08.985393310 (213h46m13.919847443s) [noise=true , address=145.79.227.14:4840 , piece_id=4B562IFYNDJL25GTL5TVLYCAZDJCDWXUPSOW4UCRJH2SCJIXSLTA, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54136->145.79.227.14:4840: use of closed network connection
Feb 26 18:03:08.985453811 (213h46m13.919907936s) [noise=true , address=145.79.227.5:4224 , piece_id=3IBRVOY65JJUBM4KY4MINDAUU5OFPGDD5WQBWJAKMJODFSFI6IRQ, seg_pos={0 0}, upload=188070]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:145.79.227.5:4224): protocol: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39750->145.79.227.5:4224: use of closed network connection
Feb 26 18:03:08.985487049 (213h46m13.919941172s) [seg_pos={0 0}, upload=188070]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Feb 26 18:03:09.647036800 (213h46m14.581490932s) [upload=188070]: Done uploading segments...
Feb 26 18:03:09.647104855 (213h46m14.581559003s) [upload=188070]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.664153488s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [...X.....X.....X...........X..X..........XX.................] duration:
1.674111218s trace:
https://jaeger.dev.storj.io/trace/3eba5f9dcfb45a38
FAIL (2026-02-26 18:03:03 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.674129367s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
319.471687ms remote:
"tcp:45.38.24.2:443" response headers:
X-Xss-Protection=1; mode=block Date=Thu, 26 Feb 2026 18:03:01 GMT Accept-Ranges=bytes Content-Length=284 Content-Type=application/xml Server=Storj Vary=Origin Content-Security-Policy=block-all-mixed-content X-Amz-Request-Id=1897DDEB4D490A75 X-Request-Id=UN2JsJbuMB8 target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/697df50c124e029d
FAIL (2026-02-26 18:03:01 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DDEB4D490A75</RequestId><HostId></HostId></Error>"
Duration: 319.515635ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
9.000195ms remote:
"tcp:185.244.226.3:443" response headers:
Accept-Ranges=bytes Content-Length=274 Content-Type=application/xml X-Request-Id=SomGZ7fGrHw Date=Thu, 26 Feb 2026 18:02:55 GMT Content-Security-Policy=block-all-mixed-content Server=Storj Vary=Origin X-Amz-Request-Id=1897DDE9F6E2CECF X-Xss-Protection=1; mode=block target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/4b7bf36d350bfdfb
FAIL (2026-02-26 18:02:55 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DDE9F6E2CECF</RequestId><HostId></HostId></Error>"
Duration: 9.025828ms
edge
gateway
vhost
bgp
eu1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
14.45602ms remote:
"tcp:185.244.226.3:443" response headers:
Accept-Ranges=bytes Content-Length=274 Content-Type=application/xml X-Amz-Request-Id=1897DDE723A87B3F X-Request-Id=WeGwKrpbizR Content-Security-Policy=block-all-mixed-content Server=Storj Vary=Origin X-Xss-Protection=1; mode=block Date=Thu, 26 Feb 2026 18:02:43 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/3fcbb09143e72d3a
FAIL (2026-02-26 18:02:43 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DDE723A87B3F</RequestId><HostId></HostId></Error>"
Duration: 14.482876ms
edge
gateway
vhost
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
304.914831ms remote:
"tcp:45.38.24.2:443" response headers:
X-Xss-Protection=1; mode=block Date=Thu, 26 Feb 2026 18:02:39 GMT Accept-Ranges=bytes Server=Storj X-Amz-Request-Id=1897DDE637D62760 X-Request-Id=yEHiWG2SZv Content-Length=284 Content-Security-Policy=block-all-mixed-content Content-Type=application/xml Vary=Origin target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/559b778e5a482725
FAIL (2026-02-26 18:02:39 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DDE637D62760</RequestId><HostId></HostId></Error>"
Duration: 304.945989ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [....X.....X.....X...........X..X..........XX................] duration:
1.650349439s trace:
https://jaeger.dev.storj.io/trace/3eba5f9dcfb45a38
FAIL (2026-02-26 18:02:32 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.65038033s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
9.586084ms remote:
"tcp:185.244.226.3:443" response headers:
Accept-Ranges=bytes Content-Length=274 Content-Security-Policy=block-all-mixed-content Content-Type=application/xml Vary=Origin X-Request-Id=bXS2ywHp45q Date=Thu, 26 Feb 2026 18:02:31 GMT Server=Storj X-Amz-Request-Id=1897DDE47A92BFEC X-Xss-Protection=1; mode=block target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/2142cb39fcc0121f
FAIL (2026-02-26 18:02:31 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897DDE47A92BFEC</RequestId><HostId></HostId></Error>"
Duration: 9.615584ms
edge
gateway
vhost
bgp
eu1
|