…#2824)

## What ❔

Passthrough env vars to the SGX enclave.

Relevant logs showcasing the issue:
https://grafana.matterlabs.dev/goto/1iFHMIeIg?orgId=1

## Why ❔

To fix the bug.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

## What ❔
Add setup keys step to prover init

…s#2800)

## What ❔

Add `--docker` flag for `zk_inception prover run` which will allow
prover components to run from docker images.

## Why ❔

To decrease setup time/improve UX

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

## What ❔

Adds README for verified sources fetcher

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

## What ❔

Limits rayon threadpool size to 2 and tokio threadpool size to 4 in
zk_toolbox CI.
I have checked locally, and with this configuration time to run
integration tests is pretty close to the default configuration.

## Why ❔

By default, both tokio and rayon will try to use all the CPUs. When we
run multiple Rust binaries at the same time (3 servers and 3 ENs in our
case), it causes a lot of conflict for resources, regardless of the
number of CPUs.

Extraction of the attester committee from consensus registry state. If
consensus registry address is not specified, we fall back to attester
committee from genesis. This pr does NOT enable the dynamic attestation,
as the registry address needs to be added to the main node config first.

---------

Co-authored-by: Moshe Shababo <17073733+moshababo@users.noreply.github.com>
Co-authored-by: Igor Aleksanov <popzxc@yandex.ru>

## What ❔

- Removes a data race from the revert integration test (a single node
one); a batch gets executed after it is verified that not all batches
are executed.
- Removes copy-paste between single-node and EN tests.
- Structures revert tests into smaller steps.

## Why ❔

Makes the test less flaky and easier to maintain.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

…atter-labs#2804)

## What ❔

- Fixes the "missing revert data" error by updating the used reth Docker
image. The error is probably caused by [this
issue](paradigmxyz/reth#7381) fixed in the new
reth versions.
- Removes "web3 API compatibility tests › Should check API returns error
when there are too many logs in eth_getLogs" test as fundamentally flaky
and able to poison other tests.
- Adds logging for upgrade test to investigate L1 "nonce too low"
errors.

## Why ❔

Flaky CI bad.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

…labs#2806)

## What ❔

Extracts oneshot VM executor to the executor crate.

## Why ❔

To make executor logic more reusable and maintainable.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

…abs#2836)

## What ❔

- Fixes DB connection starvation during snapshot recovery. Caused by the
insufficient number of connections in the DB pool provided to recovery
logic.
- Additionally, fixes max concurrency of recovery not being set.

## Why ❔

Connection starvation errors degrade UX.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

When a method is missing:
proxyd returns HTTP 403:  methodnotfound
while api server returns HTTP 200: methodnotfound

we need to handle both.

)

"{err}" is not useful, because it truncates the stack trace of anyhow
errors.

`zks t i` now accepts a `-t` flag that can can be used to specify a
pattern. Only matching tests are run.

## What ❔

Resolves matter-labs#2783

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

…r-labs#2830)

## What

- Removes `locate_workspace` and `workspace_dir_or_current_dir` methods.
- Instead, introduces `Workspace` type that is aware of different Cargo
workspaces in the codebase.

## Why

The approach with a single `locate_workspace` doesn't work well for our
codebase, since we have multiple workspaces. It resulted in some very
implicit and convoluted code (see the removed `get_base_dir` in prover
workspace).

New approach handles all 3 workspaces _plus_ the lack of a workspace.

## What ❔

Add prover subcommand for `zk_supervisor`. Add the following subcommand:
* `zk_supervisor prover info` - Prints information about current prover
setup.
* `zk_supervisor prover insert-version` - Insert new protocol version in
prover database(integration with `prover_cli`).
* `zk_supervisor prover insert-batch` - Insert new batch in prover
database(integration with `prover_cli`).

Add automatic creation of `prover/artifacts/witness_inputs` dirs if the
storage is file backed on init.

## Why ❔

To improve UX of working with provers.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

## What ❔

<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

Signed-off-by: Danil <deniallugo@gmail.com>

## What ❔

* Reduced uncompressed size from 7.5GB to 4.81 GB.
* Changes to make zk_environment image smaller
* compile solc with release mode
* remove unnecessary components from google cli
* build things first, and then only copy the artifacts.

## Why ❔

* This image is used on ALL of our CI builds - so reducing its size will
speed up the CIs.

## What ❔

Fix getting chain config in zk_supervisor prover
Fix artifacts path when initializing.
Setup data path for provers

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

## What ❔

`cargo install --list` doesn't work properly in CI because we copy only
binaries to zk-environment
also increase reth block-time and fixes upgrade-test which improves
tests stability

## Why ❔

we rely on cargo-nextest in tests

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

## What ❔

This PR changes the approach to managing 3rd party DA clients. It was
assumed before that they will be stored in a separate repository
(hyperchain-da), but to simplify the processes and solve a recursive
dependency problem, we decided to manage those within `zksync-era`.

The config now defines which DA client will be used, for proto-based
configuration it requires adding these lines to general.yaml:
```
da_client:
  avail:
    api_node_url: wss://turing-rpc.avail.so/ws
    bridge_api_url: undefined
    seed: SEED_PHRASE
    app_id: 82
    timeout: 3
    max_retries: 5
``` 
for env-based:
```
    DA_CLIENT="Avail"
    DA_API_NODE_URL="localhost:12345"
    DA_BRIDGE_API_URL="localhost:54321"
    DA_SEED="SEED_PHRASE"
    DA_APP_ID=1
    DA_TIMEOUT=2
    DA_MAX_RETRIES=3
```
If no config is provided - the default behavior is to use NoDA client
(same as now).

The `da_client` config might be merged with `da_dispatcher` at some
point as the second depends on the first one, so their separation does
not make much sense (apart from simplification of the configs). But I'd
prefer to do it as a separate PR in case we decide to merge them.

The client was reimplemented using only lightweight libraries from
crates.io, so it doesn't have any visible impact on build time.

## Why ❔

To enable seamless integration with 3rd party DA clients in
`zksync-era`.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

## What ❔

Support legacy bridges in zk toolbox 

## Why ❔

We still have some legacy bridges in production, so we have to have an
opportunity to test it

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

---------

Signed-off-by: Danil <deniallugo@gmail.com>

## What ❔
Add external_node consensus support

---------

Signed-off-by: Danil <deniallugo@gmail.com>
Co-authored-by: Danil <deniallugo@gmail.com>

## What ❔

* A small tool to generate the selector hashes based on the ABI from
json files

## Why ❔

* The output json can be useful for humans to better understand some of
the errors (and calldata)
* It can also be read by our tools, to make the debugging easier.

In the future, we could call this tool regularly on each contracts
version change, but for now it can stay as manual.

## What ❔

- Loads setup keys to memory in parallel (for GPU prover only).
- Refactors a bunch of related code for simplicity.

## Why ❔

- Locally I've observed load time going from ~30s to ~12s, so ~60%
improvement for prover start time.
- Readability & maintainability.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

## What ❔

- Use latest versions of crypto, protocol, gpu, and consensus crates.
- Remove solved cargo deny advisories from the allowlist.

## Why ❔

- A bunch of fixes/improvements were done.
- Optimization of dependency graph.
- Solving cargo deny advisories.

## What ❔

Introduce new structure for witness generators.
Introduce `ArtifactsManager` trait responsible for operations with
object store and artifacts.

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.