Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Huggingface downloader & Simpler log message & InterruptMixin #2

Merged
merged 4 commits into from
Aug 2, 2023
Merged

Conversation

c0sogi
Copy link
Owner

@c0sogi c0sogi commented Aug 2, 2023

This pull request encompasses several enhancements to usability and code refactoring. The primary changes include:

  1. Automatic Model Downloader: In our previous implementation, the model_path attribute in model_definitions.py required an actual filename of a model. We have now upgraded this to accept the name of a HuggingFace repository instead. As a result, the specified model is automatically downloaded when needed. For instance, if you define TheBloke/NewHope-GPTQ as the model_path, the necessary files will be downloaded into models/gptq/thebloke_newhope_gptq. This functionality works similarly for GGML.

  2. Simpler Log Message: We've made our log messages more concise when using Completions, Chat Completions, or Embeddings endpoints. These logs will now fundamentally display elapsed time, token usage, and token generations per second.

  3. Improved Responsiveness for Job Cancellation: The Event object in SyncManager now sends an interrupt signal to worker processes. It checks the is_interrupted property at the most low-level accessible area and tries to cancel the operation.

These changes foster more intuitive use of our application and enhance its overall responsiveness. They streamline model handling by allowing automatic downloads from a repository, rather than relying on specific file names. The job cancellation process is now more reactive, potentially saving computing resources and time if a process needs to be halted. Finally, our log messages are now cleaner and more informative, providing essential information for monitoring performance and usage.

@c0sogi c0sogi changed the title Use Hugginface model downloader & Add InterruptMixin Huggingface downloader & Simpler log message & InterruptMixin Aug 2, 2023
@c0sogi c0sogi merged commit 344ab12 into master Aug 2, 2023
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant