Testing ember v7+v8 support in dev branch/edge or 1.39.1+ release

[Nerivec]

Edit: This is now available in 1.39.1 release.

Dev branch (edge add-on) now supports both EmberZNet v7 and v8 (zigbee-herdsman 0.51+). It introduces a lot of changes (which have been applied to v7 as well).
We need some feedback from dev testers! 😉

Please do not report known issues, unless they behave differently, or have been fixed with the new dev branch.

Make sure to specify which version you are testing the dev branch with (v7 or v8).

For v8, you can try one of these firmware. Please note these are preliminary, and remain vastly untested on most adapters at the moment. And also, flashers that don't support v8 yet, will likely not allow you to revert back to v7 after flashing v8 unless you enter the bootloader manually. You can try https://github.com/Nerivec/ember-zli which uses the latest zigbee-herdsman under the hood (please report any issue with this new tool in the repository directly, not here).

Things you will undoubtedly notice:

• [v7-v8] Driver-level error/status codes have changed (slightly or completely depending on the place) due to a complete revamp of the EmberZNet firmware.
• [v7-v8] Startup sequence no longer assigns some firmware values that were causing troubles due to changes in recent EmberZNet versions. As a result, the startup sequence will look quite a bit shorter in the logs.
• [v7-v8] Some logging has been removed/altered to remove sensitive information from log files.
• [v8] Support for Silabs chips series 0 and 1 has been removed in new firmware. If you have an older adapter that uses one of these two series, you will not see a v8 firmware for it, and will have to remain to v7, or get a newer model adapter.
  • Note: MG24-based adapters are still not widely available, but from all appearances, they do offer a nice upgrade from previous models, so you may want to look into these if you find yourself in that situation.

Known issues with v8 firmware:

• LQI values are broken in devices list/page (ok in network map). The firmware appears to not be reporting the values properly. - Issue reported to Silabs.
• [Appears fixed in 8.0.1] Stability issues in networks under stress (networks with lots of spamming, poorly implemented devices, etc.)

Silabs v8 release notes: https://www.silabs.com/documents/public/release-notes/emberznet-release-notes-8.0.1.0.pdf

[JuanDYB]

MG21 chips are not recommended? I have seen the SLZB-06M which uses the EFR32MG21 chip model. There is a model from SMLight with EFR32MG24 ( SLZB-06Mg24) but at least in spain I have not seen in any place.

[Nerivec]

Unfortunately MG24 adapters are still rare indeed. As previously stated, they will offer a nice specs upgrade (2-4 times!), for anyone looking to upgrade in the near future. Until then, MG21 remains the most widely available option.
https://www.zigbee2mqtt.io/guide/adapters/emberznet.html#recommended

[JuanDYB]

So, in your opinnion maybe it's better to wait for mg24? I don't know if mg21 will be well supported when mg24 are more extended in the market ..
With regards,

[Hedda]

There is no need to wait as a later migration from EFR32MG21 to EFR32MG24 or EFR32MG26 should not be any problems in theory.

The firmware is the same and EFR32MG21 can handle up to around 200 Zigbee 3.0 devices, so really need EFR32MG24 or EFR32MG26 once you start adding more Zigbee 3.0 devices than 200, (you can add as many older Zigbee devices as you like regardless as they do not have the same security overhead and thus do not require the additional RAM and CPU resources that are provided in the EFR32MG24 MCU).

Newer EFR32MG24 or EFR32MG26 might technically or at least on paper provide other radio benefits and can be a little less sensitive to noise, but you should never rely on the Zigbee Coordinator alone and instead make sure to add loads of Zigbee Router devices regardless.

[Hedda]

For developer reference, EZSP protocol version is incremented to EZSP v14 in EmberZNet 8.0.0.0 firmware.

EmberZNet 8.0.0.0 brings support for EFR32MG26 (and MGM260) also known as xG26 and MG26 as well.

The firmware is not also built from the new Simplicity SDK, so no longer built from the "old" Gecko SDK.

• More information and links here -> Silabs Zigbee EmberZNet 8.0.0.0 (and Silicon Labs Gecko SDK 5.0.0.0) announced and to be released in Q3 2024 zigbee-herdsman#1093

[chris-1243]

@Nerivec

I finally decided to flash your latest firmware on a Dongle-E for testing (v8.0.1.0).

I paired easily an Ikea led and I had a try with a PTM216Z laying around (which brings me crazy as well for NFC config....).

Here is what I tried with my ZGP device:

• Direct pairing to the coordinator as ZGP7 (seen as PTM216Z): working;
• Direct pairing to the coordinator as ZGP2 (I did modify the config via NFC and it is seen as a PTM215ZE): working, at least like in zstack regarding the commands sent;
• Direct pairing to the Ikea bulb as ZGP7 (seen as PTM216Z): not working. This was expected as this light is not able to translate ZGP7 correctly like in zstack;
• Direct pairing to the Ikea bulb as ZGP2 (seen as PTM215ZE): working.

I hope it will help you. I had to revert to 7.4.3.0 as the software I used on Windows to sniff traffic is not updated yet for EZSP v14. The flasher from @darkxst https://github.com/darkxst/silabs-firmware-builder made a great job in updating and downgrading without any issues.

A log file of my different tries in debug mode log.txt

[Nerivec]

Thanks for the feedback. Looks like GP is on the right track!

PS: Check out https://github.com/Nerivec/ember-zli/wiki/Sniff if you need a 8.0.x-compatible sniffer (it uses zigbee-herdsman under the hood, same as Z2M).

[chris-1243]

PS: Check out https://github.com/Nerivec/ember-zli/wiki/Sniff if you need a 8.0.x-compatible sniffer (it uses zigbee-herdsman under the hood, same as Z2M).

I had a quick check and your tool looks great. May I use it on a windows 11 based computer ? (Edit) Sorry, I was too lazy to search the web and the Zigbee2mqtt documentation

Your job is just incredible...thanks

Definitely, it is too late to read correctly

https://github.com/Nerivec/ember-zli/wiki

[Nerivec]

Tarballs offer portability, and "no-setup" for one-offs, but if you are planning to use it regularly, install nodejs (tested with win 11 + node v20), it's easier to update packages with npm, and ember-zli will be in the PATH, so, can execute from anywhere with ease 😉.

[Nerivec]

@Hellou1337 @lawrencedudley @0BanZai0
I built some 8.0.1 firmware with higher config values (untested), if you want to test them, pick "max" version here.

Make sure to read the first post here, as there are still some caveats with 8.0.1 (like broken LQI).
If you encounter any issue with these firmware, please report them in the repo, not here. Same for feedback 😉

[Nerivec]

@Hellou1337 Ideally, you want to optimize groups for efficiency. Depending on the automation behind them, sometimes, a group makes more sense, sometimes, you are better off without. You want to avoid scenario where many broadcasts get sent in a short time period (this usually happens with dimmers and such devices when they lack proper throttling mechanisms).

[Hellou1337]

Yea, I kinda tried to do that. Basically I have for each room in my house one group and thats it. Sometimes there are only 2 lights per group (in small rooms) and those I could do without Z2M groups. In every room is motion sensor that switch the light on or then off (or sometimes just monitors without any real automation). And that it basically is. No dimmer or switches (well one GP switch in basement) or anything that could overload the network. I had one dimmer but it was exactly as you said - it kinda locked up the network when pushed too fast. So for safety reasons it is not used for now.
There might be some unnecessary commands to the groups - like sending off commands to lights even though they are off (ex. during the day), but should not be overwhelming. However when wife and kids are all over the place, there could be alot of data from every corner of the house 😂
Nearly the whole network is done with philips lights and sensors and been rock solid until recently even on "old" ezsp with zbdongle-e 6.10. firmware. Sadly for me I do prefer the way "new is always better" and not the "dont fix it if it isnt broken"...
Will report soon with the "max" firmware.

[Hellou1337]

Anyway,
tried to flash the 8.0.1.0 max firmware and the flasher ends up with error.
Can flash 8.0.0.0, but not any 8.0.1.0 or 8.0.1.0

Logs

s6-rc: info: service banner successfully started
s6-rc: info: service universal-silabs-flasher: starting
[10:28:56] INFO: Starting universal-silabs-flasher with /dev/ttyACM0 (bootloader baudrate 115200)
Traceback (most recent call last):
  File "/usr/local/bin/universal-silabs-flasher", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/universal_silabs_flasher/flash.py", line 40, in inner
    return asyncio.run(f(*args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/universal_silabs_flasher/flash.py", line 297, in flash
    metadata = fw_image.get_nabucasa_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/universal_silabs_flasher/firmware.py", line 178, in get_nabucasa_metadata
    return NabuCasaMetadata.from_json(json.loads(metadata))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/universal_silabs_flasher/firmware.py", line 111, in from_json
    fw_type = FirmwareImageType(fw_type)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/enum.py", line 714, in __call__
    return cls.__new__(cls, value)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/enum.py", line 1137, in __new__
    raise ve_exc
ValueError: 'ncp-uart-sw' is not a valid FirmwareImageType

[Nerivec]

Try using ember-zli instead (link in first post).

[Hellou1337]

So i tried the max firmware with rebuilding the z2m groups as it was.
And - i have zero problems and errors since then.

[chris-1243]

It might not be the right place to post.

I am wondering how ember is managing the IEEE address while restoring a backup. Especially while changing the coordinator.

I asked there

darkxst/silabs-firmware-builder#120 (comment)

and according to this thread, the IEEE address is stored as a token in NV memory. This means, it would be not needed to write it while flashing a firmware.

Second question arises... What about zstack or other driver ? This would remove the pain of re-pairing everything and users would have the possibility to move from one driver to another depending on their needs...

[Nerivec]

z2m ember currently is not messing with the adapter's IEEE (only backup/restore network info). If I'm not mistaken, the ability to manipulate tokens depends on the firmware being compiled with the tokens component, which may or may not be the case depending on the options set by the builder. It would result in varying behavior depending on adapter+firmware. This would add a bit too much randomness to the process I think (especially since all this happens in the background, without any menu or anything, and only logs for feedback). I'll look into it to confirm and see what could be done in a more reliable fashion.

ember-zli should be able to do it via tokens menus though (can backup/restore the entire set of zigbee tokens in nvm3), assuming your firmware supports tokens manipulation.

[Xploder]

Yes I've tried restoring just the network via ember-zli from an ZBDongleE to SLZB-06M and it didn't work. After also restoring NVM3 tokens with the same tool I was back up immediately - very smooth migration!

[chris-1243]

@Nerivec

In order to get a better support of ZGP frames/devices, would you accept to receive one or more PTM device directly from Enocean according this thread ?