Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compilation speed #175

Open
beadon opened this issue Jul 30, 2023 · 8 comments
Open

Compilation speed #175

beadon opened this issue Jul 30, 2023 · 8 comments

Comments

@beadon
Copy link
Contributor

beadon commented Jul 30, 2023

With the recent issue related to building, I was reminded of the time it took you to download and compile the required sources.

For kicks I wanted to see how far apart we were for compilation performance.

Timing the local compilation I am downloading and building in less than 7 seconds. You had mentioned it was taking "minutes" to do the same from a clean repo oull. What could possibly be taking so long on your side ?

reference 'dumb' compilation test:

beadon@semiauto:~/build/purple-gowhatsapp$ time ( mkdir build && cd build && cmake .. && cmake --build . )
-- The C compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Checking for module 'purple'
--   Found purple, version 2.14.12
-- PURPLE_INCLUDE_DIRS: /usr/include/libpurple;/usr/include/glib-2.0;/usr/lib/x86_64-linux-gnu/glib-2.0/include
-- PURPLE_LIBRARY_DIRS: /usr/lib/x86_64-linux-gnu
-- PURPLE_LIBRARIES: purple;glib-2.0
-- PURPLE_PLUGIN_DIR: /usr/lib/x86_64-linux-gnu/purple-2
-- PURPLE_DATA_DIR: /usr/share
-- Checking for module 'gdk-pixbuf-2.0'
--   Found gdk-pixbuf-2.0, version 2.42.10
-- Checking for module 'opusfile'
--   Found opusfile, version 0.12
-- PIXBUF_INCLUDE_DIRS: /usr/include/gdk-pixbuf-2.0;/usr/include/glib-2.0;/usr/lib/x86_64-linux-gnu/glib-2.0/include;/usr/include/libpng16;/usr/include/x86_64-linux-gnu;/usr/include/libmount;/usr/include/blkid
-- OPUSFILE_INCLUDE_DIRS: /usr/include/opus
-- OPUSFILE_LIBRARIES: opusfile
-- Configuring done
-- Generating done
-- Build files have been written to: /home/beadon/build/purple-gowhatsapp/build
[  4%] Building C object src/go/CMakeFiles/purple-dummy.dir/purple-dummy.c.o
[  8%] Building C object src/go/CMakeFiles/purple-dummy.dir/__/c/constants.c.o
[ 12%] Linking C static library libpurple-dummy.a
[ 12%] Built target purple-dummy
[ 16%] Generating purple-go-whatsapp.a, purple-go-whatsapp.h
go: downloading github.com/alfg/mp4 v0.0.0-20210728035756-55ea58c08aeb
go: downloading github.com/go-sql-driver/mysql v1.6.0
go: downloading github.com/lib/pq v1.10.4
go: downloading github.com/mattn/go-sqlite3 v1.14.10
go: downloading github.com/mdp/qrterminal/v3 v3.0.0
go: downloading github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
go: downloading go.mau.fi/whatsmeow v0.0.0-20230727120640-c313a80ab292
go: downloading golang.org/x/net v0.9.0
go: downloading google.golang.org/protobuf v1.30.0
go: downloading rsc.io/qr v0.2.0
go: downloading go.mau.fi/libsignal v0.1.0
go: downloading golang.org/x/crypto v0.8.0
go: downloading filippo.io/edwards25519 v1.0.0
go: downloading github.com/gorilla/websocket v1.5.0
go: downloading golang.org/x/text v0.9.0
[ 20%] Built target purple-go-whatsapp
[ 25%] Building C object src/c/CMakeFiles/whatsmeow.dir/init.c.o
[ 29%] Building C object src/c/CMakeFiles/whatsmeow.dir/login.c.o
[ 33%] Building C object src/c/CMakeFiles/whatsmeow.dir/qrcode.c.o
[ 37%] Building C object src/c/CMakeFiles/whatsmeow.dir/bridge.c.o
[ 41%] Building C object src/c/CMakeFiles/whatsmeow.dir/process_message.c.o
[ 45%] Building C object src/c/CMakeFiles/whatsmeow.dir/display_message.c.o
[ 50%] Building C object src/c/CMakeFiles/whatsmeow.dir/groups.c.o
[ 54%] Building C object src/c/CMakeFiles/whatsmeow.dir/blist.c.o
[ 58%] Building C object src/c/CMakeFiles/whatsmeow.dir/constants.c.o
[ 62%] Building C object src/c/CMakeFiles/whatsmeow.dir/send_message.c.o
[ 66%] Building C object src/c/CMakeFiles/whatsmeow.dir/handle_attachment.c.o
[ 70%] Building C object src/c/CMakeFiles/whatsmeow.dir/send_file.c.o
[ 75%] Building C object src/c/CMakeFiles/whatsmeow.dir/presence.c.o
[ 79%] Building C object src/c/CMakeFiles/whatsmeow.dir/options.c.o
[ 83%] Building C object src/c/CMakeFiles/whatsmeow.dir/receipt.c.o
[ 87%] Building C object src/c/CMakeFiles/whatsmeow.dir/pixbuf.c.o
[ 91%] Building C object src/c/CMakeFiles/whatsmeow.dir/commands.c.o
[ 95%] Building C object src/c/CMakeFiles/whatsmeow.dir/opusreader.c.o
[100%] Linking C shared library libwhatsmeow.so
[100%] Built target whatsmeow

real	0m6.894s
user	0m4.383s
sys	0m1.149s
@beadon beadon changed the title Complication speed Compilation speed Jul 30, 2023
@beadon
Copy link
Contributor Author

beadon commented Jul 31, 2023

Just for kicks, even faster ...

beadon@semiauto:~/build/purple-gowhatsapp$ time ( mkdir build && cd build && cmake .. && cmake --build . -j4 )
-- The C compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Checking for module 'purple'
--   Found purple, version 2.14.12
-- PURPLE_INCLUDE_DIRS: /usr/include/libpurple;/usr/include/glib-2.0;/usr/lib/x86_64-linux-gnu/glib-2.0/include
-- PURPLE_LIBRARY_DIRS: /usr/lib/x86_64-linux-gnu
-- PURPLE_LIBRARIES: purple;glib-2.0
-- PURPLE_PLUGIN_DIR: /usr/lib/x86_64-linux-gnu/purple-2
-- PURPLE_DATA_DIR: /usr/share
-- Checking for module 'gdk-pixbuf-2.0'
--   Found gdk-pixbuf-2.0, version 2.42.10
-- Checking for module 'opusfile'
--   Found opusfile, version 0.12
-- PIXBUF_INCLUDE_DIRS: /usr/include/gdk-pixbuf-2.0;/usr/include/glib-2.0;/usr/lib/x86_64-linux-gnu/glib-2.0/include;/usr/include/libpng16;/usr/include/x86_64-linux-gnu;/usr/include/libmount;/usr/include/blkid
-- OPUSFILE_INCLUDE_DIRS: /usr/include/opus
-- OPUSFILE_LIBRARIES: opusfile
-- Configuring done
-- Generating done
-- Build files have been written to: /home/beadon/build/purple-gowhatsapp/build
[  8%] Building C object src/go/CMakeFiles/purple-dummy.dir/purple-dummy.c.o
[  8%] Building C object src/go/CMakeFiles/purple-dummy.dir/__/c/constants.c.o
[ 12%] Linking C static library libpurple-dummy.a
[ 12%] Built target purple-dummy
[ 16%] Generating purple-go-whatsapp.a, purple-go-whatsapp.h
go: downloading github.com/alfg/mp4 v0.0.0-20210728035756-55ea58c08aeb
go: downloading github.com/go-sql-driver/mysql v1.6.0
go: downloading github.com/lib/pq v1.10.4
go: downloading github.com/mattn/go-sqlite3 v1.14.10
go: downloading github.com/mdp/qrterminal/v3 v3.0.0
go: downloading github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
go: downloading go.mau.fi/whatsmeow v0.0.0-20230727120640-c313a80ab292
go: downloading golang.org/x/net v0.9.0
go: downloading google.golang.org/protobuf v1.30.0
go: downloading rsc.io/qr v0.2.0
go: downloading go.mau.fi/libsignal v0.1.0
go: downloading golang.org/x/crypto v0.8.0
go: downloading filippo.io/edwards25519 v1.0.0
go: downloading github.com/gorilla/websocket v1.5.0
go: downloading golang.org/x/text v0.9.0
[ 20%] Built target purple-go-whatsapp
[ 37%] Building C object src/c/CMakeFiles/whatsmeow.dir/init.c.o
[ 37%] Building C object src/c/CMakeFiles/whatsmeow.dir/login.c.o
[ 37%] Building C object src/c/CMakeFiles/whatsmeow.dir/qrcode.c.o
[ 37%] Building C object src/c/CMakeFiles/whatsmeow.dir/bridge.c.o
[ 54%] Building C object src/c/CMakeFiles/whatsmeow.dir/groups.c.o
[ 54%] Building C object src/c/CMakeFiles/whatsmeow.dir/blist.c.o
[ 54%] Building C object src/c/CMakeFiles/whatsmeow.dir/process_message.c.o
[ 54%] Building C object src/c/CMakeFiles/whatsmeow.dir/display_message.c.o
[ 58%] Building C object src/c/CMakeFiles/whatsmeow.dir/constants.c.o
[ 62%] Building C object src/c/CMakeFiles/whatsmeow.dir/send_message.c.o
[ 66%] Building C object src/c/CMakeFiles/whatsmeow.dir/handle_attachment.c.o
[ 70%] Building C object src/c/CMakeFiles/whatsmeow.dir/send_file.c.o
[ 75%] Building C object src/c/CMakeFiles/whatsmeow.dir/presence.c.o
[ 79%] Building C object src/c/CMakeFiles/whatsmeow.dir/options.c.o
[ 83%] Building C object src/c/CMakeFiles/whatsmeow.dir/receipt.c.o
[ 87%] Building C object src/c/CMakeFiles/whatsmeow.dir/pixbuf.c.o
[ 91%] Building C object src/c/CMakeFiles/whatsmeow.dir/commands.c.o
[ 95%] Building C object src/c/CMakeFiles/whatsmeow.dir/opusreader.c.o
[100%] Linking C shared library libwhatsmeow.so
[100%] Built target whatsmeow

real	0m3.669s
user	0m3.922s
sys	0m1.114s

sub 4 seconds ...

@hoehermann
Copy link
Owner

A couple of factors.

  • The CPU in my machine is over ten years old.
  • My internet connection allows a download rate 1.4 MB/s. That is why I do want to keep the cache. Fiber to the home is expected to become available "very soon" for the past decade. Welcome to Germany, land of progress.

This is especially pronounced for the win32 build:

$ time cmake --build .
Consolidate compiler generated dependencies of target purple-dummy
[ 12%] Built target purple-dummy
[ 16%] Generating purple-go-whatsapp.a, purple-go-whatsapp.h
go: downloading github.com/alfg/mp4 v0.0.0-20210728035756-55ea58c08aeb
go: downloading github.com/go-sql-driver/mysql v1.6.0
go: downloading github.com/lib/pq v1.10.4
go: downloading github.com/mattn/go-sqlite3 v1.14.10
go: downloading github.com/mdp/qrterminal/v3 v3.0.0
go: downloading github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
go: downloading go.mau.fi/whatsmeow v0.0.0-20230727120640-c313a80ab292
go: downloading golang.org/x/net v0.9.0
go: downloading google.golang.org/protobuf v1.30.0
go: downloading rsc.io/qr v0.2.0
go: downloading go.mau.fi/libsignal v0.1.0
go: downloading golang.org/x/crypto v0.8.0
go: downloading filippo.io/edwards25519 v1.0.0
go: downloading github.com/gorilla/websocket v1.5.0
go: downloading golang.org/x/text v0.9.0
[ 20%] Built target purple-go-whatsapp
Consolidate compiler generated dependencies of target whatsmeow
[ 25%] Building C object src/c/CMakeFiles/whatsmeow.dir/init.c.obj
[ 29%] Building C object src/c/CMakeFiles/whatsmeow.dir/login.c.obj
[ 33%] Building C object src/c/CMakeFiles/whatsmeow.dir/process_message.c.obj
[ 37%] Building C object src/c/CMakeFiles/whatsmeow.dir/groups.c.obj
[ 41%] Building C object src/c/CMakeFiles/whatsmeow.dir/blist.c.obj
[ 45%] Building C object src/c/CMakeFiles/whatsmeow.dir/send_message.c.obj
[ 50%] Building C object src/c/CMakeFiles/whatsmeow.dir/send_file.c.obj
[ 54%] Building C object src/c/CMakeFiles/whatsmeow.dir/presence.c.obj
[ 58%] Building C object src/c/CMakeFiles/whatsmeow.dir/receipt.c.obj
[ 62%] Building C object src/c/CMakeFiles/whatsmeow.dir/commands.c.obj
[ 66%] Linking C shared library libwhatsmeow.dll
[100%] Built target whatsmeow

real	1m41,724s
user	0m0,012s
sys	0m0,046s

@beadon
Copy link
Contributor Author

beadon commented Jul 31, 2023

/I suspect the CPU, as you do, but I suspect your storage is the real slowdown. Given the age of the hardware and the ready availability of components it's cheap to update the CPU and the storage relatively easily.

Here's that fastest processor for that socket -- should fit, give a nice performance boost, about $29 USD (free shipping): https://www.ebay.com/sch/i.html?_from=R40&_trksid=p4432023.m570.l1313&_nkw=i7+3770&_sacat=0

Next up the storage - likely SATA, your primary drive likely is short on the available IOPS. About $58 USD for this 1TB , very speedy, SSD:
https://www.ebay.com/itm/354850276245

If you want to really go fast, then a M.2 to PCIE adaptor will let you step up to the next level , ~$6 :
https://www.ebay.com/itm/292931456826

And then select a M.2 drive, which can be had for $35 -- prices are recently falling dramatically.

Networking --
1.4 MB/s works out to about 11 mbps ( very close to the 10mbps of a physical link problem), so, if you sure this is your ISP speed is correct don't worry. But, your downloads there are quite small, I will need to see how much space is truly required, this should still be only a few seconds. If you are not getting this speed from the ISP, and actually have something higher, then it's time to look at ethernet cables :)

I hope these will help. For a few dollars your performance will be greatly improved. I anticipate you may even be able to find better deals locally.

@hoehermann
Copy link
Owner

Thank you for the suggestions.

Those prices reflect on the US market. In Europe, prices did not really go down. I blame the pandemic and the Russo-Ukraine war. With an international purchase, I would need to pay import tax. By the way, the CPU you recommended would only give me a slight boost, according to userbenchmark.com. The time I would spend on the upgrade process would probably exceed the benefits. Also, more and more USB-ports on my system's main-board are failing. I will probably update some day anyways. ;)

However, not everyone can do. While working on this project for the past years, I communicated with users from Africa and South America. They are stuck with 32bit systems, sometimes of the Pentium 4 era. They depend on projects like this since they cannot use the official WhatsApp Destkop client due to the limitations of their systems. I see no reason to make their life even harder by forcing a cache wipe into the build process. That was – if I remember correctly – the context I mentioned the long build times in. :)

@EionRobb
Copy link

EionRobb commented Aug 1, 2023

Might go even faster if you don't build to the intermediate obj/o files and go straight to dll/so by compiling all the c files at once (can get the benefit of not having to scan through all the header includes for multiple times, for each c file).

Speeds up compile times by about x2 for some of the other pidgin plugins I worked on, hopefully gives a boost here too?

@hoehermann
Copy link
Owner

hoehermann commented Aug 1, 2023

On my system, recompiling the plug-in's go part takes 6 seconds on my system. Recompiling the entire C part takes 2 seconds. I have no intention to speed that up.

In a previous contribution #166, Bryant opted to drop the build cache prior to building. This will make go re-download and re-compile all dependencies. That I deem unacceptable.

On a side-note: I just noticed that when using CGO (go's ffi features which we have to use), the go build process is single-threaded. According to this thread, it seems to be by design. More cores will not help in this matter.

@beadon
Copy link
Contributor Author

beadon commented Aug 1, 2023

The goal is within this thread only : Improve a developer's workflow, based on the metric of compilation speed. Specifically yours!

I am not pulling any other threads in here.

It's good to know about the 32-bit users in other countries. I would recommend then that we keep some kind of 32-bit compilation around to support these users.

I was identifying key parts of your workflow that would speed up this compilation, and probably really improve anything else you do with the machine. I was citing costs because I wasn't necessarily expecting you to buy things, but perhaps the community could ? (these are relatively modest parts, some people might even have them around)

Thoughts ?

@beadon
Copy link
Contributor Author

beadon commented Aug 1, 2023

A quick note on multi-threading the compile, notice the speedup I applied in the second try "cmake --buld . -j4 " not because I thought it should be applied, but just because that was proof the ~7s could still be improved if necessary (and as expected with the right parallelization).

That being said, on rereading your last comment -- believe I am reading that you are quoting the go downloading parts to your machine at approximately 1min 29s? ( other parts of the build about 7s total ? ).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants